From owner-freebsd-hackers@freebsd.org Sun Feb 21 00:05:07 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 49AB753D030 for ; Sun, 21 Feb 2021 00:05:07 +0000 (UTC) (envelope-from chrisj@rtems.org) Received: from nsstlmta34p.bpe.bigpond.com (nsstlmta34p.bpe.bigpond.com [203.38.21.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "", Issuer "Openwave Messaging Inc." (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Djltn5TvCz4tm1 for ; Sun, 21 Feb 2021 00:05:05 +0000 (UTC) (envelope-from chrisj@rtems.org) Received: from smtp.telstra.com ([10.10.24.4]) by nsstlfep34p-svc.bpe.nexus.telstra.com.au with ESMTP id <20210221000456.KNUP12830.nsstlfep34p-svc.bpe.nexus.telstra.com.au@smtp.telstra.com>; Sun, 21 Feb 2021 11:04:56 +1100 X-RG-Spam: Unknown X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgeduledrjeelgdduhecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfupfevtfgpvffgnffuvffttedpqfgfvfenuceurghilhhouhhtmecugedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhepuffhvfhfohfkffgfgggjtgfgsehtjeertddtfeejnecuhfhrohhmpeevhhhrihhsucflohhhnhhsuceotghhrhhishhjsehrthgvmhhsrdhorhhgqeenucggtffrrghtthgvrhhnpeeuteekvdevffefteetgeehtefhtefffefgjeduveduleehgffggfehffeiffeuleenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucfkphepudefledrudeftddrvdeghedrvddttdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhephhgvlhhopehmrghilhdrtghonhhtvghmphhorhgrrhihrdhnvghtrdgruhdpihhnvghtpedufeelrddufedtrddvgeehrddvtddtpdhmrghilhhfrhhomhepoegthhhrihhsjhesrhhtvghmshdrohhrgheqpdhrtghpthhtohepoehfrhgvvggsshguqdhhrggtkhgvrhhssehfrhgvvggsshgurdhorhhgqedprhgtphhtthhopeeorhhmrggtkhhlvghmsehuohhguhgvlhhphhdrtggrqe X-RazorGate-Vade-Verdict: clean 0 X-RazorGate-Vade-Classification: clean X-RG-VS-CLASS: clean Received: from mail.contemporary.net.au (139.130.245.200) by smtp.telstra.com (5.8.420) id 5FE6CCD00FD4EFBF; Sun, 21 Feb 2021 11:04:55 +1100 Received: from [10.10.11.8] ([10.10.11.8]) (authenticated bits=0) by mail.contemporary.net.au (8.15.2/8.15.2) with ESMTPSA id 11L04qX4046725 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 21 Feb 2021 11:04:52 +1100 (EST) (envelope-from chrisj@rtems.org) Subject: Re: sys/fs/nfsclient ACCESS attributes with RTEMS From: Chris Johns To: Rick Macklem , "freebsd-hackers@freebsd.org" References: <0e14503a-ca7b-cd0f-6472-289fc9ac301b@rtems.org> <5f66e68f-2979-dd14-f8f4-ee2bed0a5f43@rtems.org> Organization: https://www.rtems.org/ Message-ID: <5140b8d9-7f98-cd7d-eac0-9f938d73d320@rtems.org> Date: Sun, 21 Feb 2021 11:04:52 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <5f66e68f-2979-dd14-f8f4-ee2bed0a5f43@rtems.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-102.9 required=2.7 tests=ALL_TRUSTED,BAYES_00, NICE_REPLY_A,USER_IN_WELCOMELIST,USER_IN_WHITELIST autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on corb.contemporary.net.au X-Rspamd-Queue-Id: 4Djltn5TvCz4tm1 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of chrisj@rtems.org has no SPF policy when checking 203.38.21.34) smtp.mailfrom=chrisj@rtems.org X-Spamd-Result: default: False [-1.22 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-0.02)[-0.024]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_LOW(-0.10)[203.38.21.34:from]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[203.38.21.34:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:1221, ipnet:203.36.0.0/14, country:AU]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[rtems.org]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[203.38.21.34:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Feb 2021 00:05:07 -0000 On 21/2/21 9:39 am, Chris Johns wrote: > On 21/2/21 3:13 am, Rick Macklem wrote: >> Chris Johns wrote: >> If you test against a FreeBSD server, you can "sysctl vfs.nfsd.debuglevel=4" and >> make it pretty chatty to help figure out what is going on. > > Thanks, I will have a look. I have isolated a potential reason. The FreeBSD > client has this in the RPC header: > > Credentials > Flavor: AUTH_UNIX (1) > Length: 32 > Stamp: 0x5e72fba7 > Machine Name: ruru > length: 4 > contents: ruru > UID: 0 > GID: 0 > Auxiliary GIDs (2) [0, 5] > GID: 0 > GID: 5 > > and my RTEMS client has: > > Credentials > Flavor: AUTH_UNIX (1) > Length: 20 > Stamp: 0x21dae501 > Machine Name: > length: 0 > contents: > UID: 65536 I have traced the error down to this piece of code in rpc [1]: if (!xdr_uint32_t(xdrs, &cred->cr_uid)) return (FALSE); if (!xdr_uint32_t(xdrs, &cred->cr_groups[0])) return (FALSE); On RTEMS uid_t is int and this creates an incompatible pointer... (gdb) p /x *(uint32_t*)&cred->cr_uid $15 = 0x10000 Assigning the uid to the local junk variable and using that fixes my perms issue and I can make a directory. Chris [1] https://cgit.freebsd.org/src/tree/sys/rpc/authunix_prot.c#n100 From owner-freebsd-hackers@freebsd.org Mon Feb 22 20:31:17 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 751D054D6ED for ; Mon, 22 Feb 2021 20:31:17 +0000 (UTC) (envelope-from core-secretary@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4Dkv392NCTz4jtK for ; Mon, 22 Feb 2021 20:31:17 +0000 (UTC) (envelope-from core-secretary@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 4FDE554D74C; Mon, 22 Feb 2021 20:31:17 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4FA0554D7EA for ; Mon, 22 Feb 2021 20:31:17 +0000 (UTC) (envelope-from core-secretary@freebsd.org) Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dkv38757Tz4jtJ for ; Mon, 22 Feb 2021 20:31:16 +0000 (UTC) (envelope-from core-secretary@freebsd.org) Received: by mail-ej1-f47.google.com with SMTP id do6so31407017ejc.3 for ; Mon, 22 Feb 2021 12:31:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:reply-to:mime-version:subject:date :message-id:cc:to; bh=3fRDphfAyEtH9wqsFaNqS++LYnuUYNFYDvYr6GZPvi4=; b=ehCYg/dRr974Q8QjgvzfDbMi8Zm7fsaWZY5LEneE8T7a7uz8Bff+WVGzIvDM+nMt33 THzTBCS4yJkzk+DTD0Tizqa+MabV/iO2h8e0c1TxxsDFQ6vR4hE30bTATfq8sl12KqQK CFAVY6ikkWVs4/z88oo1BqglWauKCKbpnJUMLOMRmjD5pyYKKC+uyZt8EFq8L0+IlLTL /SDuSDAG8asoThbveRUMtoCGesvzrA+XR4KJShpz8r6KgROGnKRoK+uM4NEIogqycYCJ GM/12D1CUTCqGQ7gSk2D3f+qsDbLHo2mr3bSNlPaWlk8ISBM1wujso1Ic8DE7m4CAKxo qaRA== X-Gm-Message-State: AOAM530oEq1hNVKtlwBvBpc7xpZ7gGxla5xQUiiP6Q3bU9cWRKTYqN/W YZXXzlDZ6b52SA9I2lXk3B2w97+L X-Google-Smtp-Source: ABdhPJyePUVKml/U99mfpHrVfzkk+bJUno5HiZpWYKhLLoC5qKgWwN6hgsrdlSKqijOEnYe06LHk8A== X-Received: by 2002:a17:906:1e50:: with SMTP id i16mr9795631ejj.466.1614025875589; Mon, 22 Feb 2021 12:31:15 -0800 (PST) Received: from mx.bofh.network (mx.bofh.network. [2001:19f0:5001:2b77:5400:2ff:fe7b:aa2c]) by smtp.gmail.com with ESMTPSA id g3sm13361131edk.75.2021.02.22.12.31.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Feb 2021 12:31:14 -0800 (PST) Received: from [IPv6:2402:54c0:ffff:ffff:5db8:43f1:c50a:131a] ( [2402:54c0:ffff:ffff:5db8:43f1:c50a:131a]) by mx.bofh.network (OpenSMTPD) with ESMTPSA id d026c320 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Mon, 22 Feb 2021 20:31:11 +0000 (UTC) From: FreeBSD Core Team Secretary Content-Type: multipart/signed; boundary="Apple-Mail=_D4ACF78A-980D-4F04-8B24-7357681DF378"; protocol="application/pgp-signature"; micalg=pgp-sha512 Reply-To: FreeBSD Core Team Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: CORE Team Office Hours Date: Tue, 23 Feb 2021 02:31:06 +0600 Message-Id: Cc: developers@freebsd.org, hackers@freebsd.org, current@freebsd.org, Deb Goodkin To: announce@freebsd.org X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4Dkv38757Tz4jtJ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2021 20:31:17 -0000 --Apple-Mail=_D4ACF78A-980D-4F04-8B24-7357681DF378 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, Following on from the effort in 2019, The FreeBSD team is arranging = another Community Survey to help shape the future of The FreeBSD = Project. The purpose of this survey is to collect quantitative data from = the public in order to help guide the project=E2=80=99s priorities and = efforts. Similar surveys have been conducted twice by the FreeBSD = Project and we are preparing for our third. Before publishing the next survey the Core Team welcomes you to attend = for a Virtual Town Hall meeting to share some of the insights of the = 2020 Community Survey, and seek advice on how the next survey should be = conducted. The virtual Town Hall will take place on the 17th of March = 2021, at 18:00 hours GMT/UTC. See https://wiki.freebsd.org/OfficeHours = for details on how to join either a live stream to watch, or an = interactive meeting to participate. If time permits the Core Team may be able to answer questions related to = the survey process.. If you have any specific questions that you would = like answered during the session you can send an email to core@ ahead of = the meeting which the Core Team will try to address. Thanks! We look forward to meeting you. Regards, Moin (bofh), with core-secretary@ hat on --Apple-Mail=_D4ACF78A-980D-4F04-8B24-7357681DF378 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEES2Tp4L3ps+zAa1xm2MjIO0nybxcFAmA0FIpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDRC NjRFOUUwQkRFOUIzRUNDMDZCNUM2NkQ4QzhDODNCNDlGMjZGMTcACgkQ2MjIO0ny bxcXBg/9Ewpi84WQWv8vd4UKs2Tct9zg7tb5hDclvJmuNVWG06biXqVEI/TNr4+i fGIkZ3wfLaZvT5w9k8Ae7zP3k+eKIT4QqP1FmJZV1L1f0PDLZPj9ZyAWGqAgEmB3 qn807T+xrZ6LA+VuFLh0D6WD8bl4KZQhdNMd42z+reQ66/OGuzYpjzLyhy6OFB95 0+R/pT2XPCl3d7egFQhxZFI9AP4zGD3Bn3PXe7d1d/Dhf2t7Ow/iQQqyIfiXxd8t t3Da89txmUU6ZRl4dMsEsinj6O8ktcBslZ2UP2x5TbCDXcKgfwRYfGvnMg8vOAnV z2vpv6v07x+PBjMngf5RqmW+S407EoSGLBaTnyaRJeAmxYyYf84keWeU+0ZxXT2f V3NUVY3D64PHnaR05ibsoiP1DA0L0UIuUEDLXuqA1Qrp6nkhFowHpg+yyiQKcgYf 5bM/KCb7cLNFBtucm7JSmDxPcqbUarBZlxAffMR2rkTgMv3V9zGa7sDB1YncohMx vJePGD0wNvgCG5DG93rk3QfhBLrpLF2/M4Q603dN1Dn2sMpH1K7xRwA1WvhL20aP QHlpK/MWZizKJbY3DjDXsSNIGecx9Af9hhFw03RKxQmyExpv+4MU2IcZ13T545mW WoKTl6eSWKIo8cMx0vcSmRrkGGRHUWhJRUkU7zlzJyyARQiu30U= =eQnL -----END PGP SIGNATURE----- --Apple-Mail=_D4ACF78A-980D-4F04-8B24-7357681DF378-- From owner-freebsd-hackers@freebsd.org Tue Feb 23 20:50:02 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 88A4F552349 for ; Tue, 23 Feb 2021 20:50:02 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlWQK697sz4bgk for ; Tue, 23 Feb 2021 20:50:01 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f52.google.com with SMTP id d9so2884417ote.12 for ; Tue, 23 Feb 2021 12:50:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=orgxXiXu6I6qcL8EAko/dQdM9hGyBL4wSxHcyHr74R0=; b=kKdY1Eh6/TDzLTyf34b0Z/9b+9feabP1WENFyRWlhrxs0oMUSFltGsIrewAjzLi0EY dEcWw0LRXa8vzZgX76sLofYSEybgnzUM0I2ufTHcf0QWOCs6frajA5xkyUwHtELpEvVJ YQMP9TopSMRcbTDHmHbviQU0w9SMh0lNNh24UgncCsKLktFbgqZMcOIl+ORcWk6CVIVL 4OJ5BCOf+Lzg/VO7/bP1MgBxM2sbGXtEQKGT6vJk3/v73DAkiE7SN/2hY1/fk0F4Fe+X Sj3ErPo+2x79B+dkLZAm70LK+nQPbmI1lwE1oXYItQtmgCsPgz3ea++VDXayZYPu1Ipj c4XQ== X-Gm-Message-State: AOAM531QpySw7zlPRIQNuriXNhQKEwk+dHZRu+bM7UuC8hSCZbrFM8FQ fiJX6PgbeiPrjc9N977FgaYNzklgn+c0cTk+HMWX51tchsxRvQ== X-Google-Smtp-Source: ABdhPJz/njLWpGwTbsjOiQkvvNnsPXh1H93rxgME2Hnw3pswJwJJouL11tTQvYEknFWWRv9KlMm1jC0MgKNfo5xaA7Y= X-Received: by 2002:a9d:3642:: with SMTP id w60mr21541398otb.18.1614113400244; Tue, 23 Feb 2021 12:50:00 -0800 (PST) MIME-Version: 1.0 From: Alan Somers Date: Tue, 23 Feb 2021 13:49:49 -0700 Message-ID: Subject: The out-of-swap killer makes poor choices To: FreeBSD Hackers X-Rspamd-Queue-Id: 4DlWQK697sz4bgk X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.52 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.00 / 15.00]; RWL_MAILSPIKE_GOOD(0.00)[209.85.210.52:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_ALL(0.00)[]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.210.52:from]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_DOM_EQ_FROM_DOM(0.00)[]; FREEFALL_USER(0.00)[asomers]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[209.85.210.52:from:127.0.2.255]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.52:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 20:50:02 -0000 To me it's always seemed like the out-of-swap killer kills the wrong process. Oh, it does the right thing with a trivial while(1) {malloc()} test program, but not with real workloads. To summarize the logic in vm_pageout_oom: * Don't kill system, protected, or killed processes * Don't kill processes with a thread that isn't running or suspended * Kill whichever process is using the most swap or swap + ram, depending on the shortage variable. On ties, kill the newest one. This algorithm probably made sense in the days when computers had much more swap than RAM. But now it leads to several problems: * It's almost guaranteed to do the wrong thing when shortage == VM_OOM_SWAPZ and there is little or no swap configured. If no swap is configured, it will kill the newest running or suspended process. If a little bit is configured, it will probably kill some idle process, like zfsd, that is swapped out because it doesn't run very often. * Even if multiple GB of swap are configured, the OOM killer is still biased towards killing idle processes when shortage == VM_OOM_SWAPZ. Most often, the process responsible for an out-of-memory condition is not idle, and is consuming large amounts of RAM. * It ignores RLIMIT_RSS. We consider that rlimit when deciding whether to move a process from RAM to swap. * The "out of swap space" kernel message doesn't specify whether the process was killed because of insufficient swap or RAM (the shortage variable) I propose the following changes: * Incorporate shortage into the "out of swap space" message. * When walking the process list, if any process exceeds its RLIMIT_RSS, choose it immediately, without bothering to compare it to older processes. * Always consider the sum of a process's RAM + swap, regardless of the shortage variable. Does this make sense? Am I missing something about shortage == VM_OOM_SWAPZ? I don't understand why you would ever want to exclude processes' RAM usage. That logic was added in revision 2025d69ba7a68a5af173007a8072c45ad797ea23, but I don't understand the rationale. -Alan From owner-freebsd-hackers@freebsd.org Tue Feb 23 21:11:38 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 75879552F9D for ; Tue, 23 Feb 2021 21:11:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlWvF2pNwz4dD1; Tue, 23 Feb 2021 21:11:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11NLBMZt028046 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 23 Feb 2021 23:11:25 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11NLBMZt028046 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11NLBMW4028045; Tue, 23 Feb 2021 23:11:22 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 23 Feb 2021 23:11:22 +0200 From: Konstantin Belousov To: Alan Somers Cc: FreeBSD Hackers Subject: Re: The out-of-swap killer makes poor choices Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DlWvF2pNwz4dD1 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [0.98 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_SPAM_MEDIUM(0.98)[0.983]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_SHORT(1.00)[1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 21:11:38 -0000 On Tue, Feb 23, 2021 at 01:49:49PM -0700, Alan Somers wrote: > To me it's always seemed like the out-of-swap killer kills the wrong > process. Oh, it does the right thing with a trivial while(1) {malloc()} > test program, but not with real workloads. To summarize the logic in > vm_pageout_oom: > > * Don't kill system, protected, or killed processes > * Don't kill processes with a thread that isn't running or suspended > * Kill whichever process is using the most swap or swap + ram, depending on > the shortage variable. On ties, kill the newest one. > > This algorithm probably made sense in the days when computers had much more > swap than RAM. But now it leads to several problems: > > * It's almost guaranteed to do the wrong thing when shortage == > VM_OOM_SWAPZ and there is little or no swap configured. If no swap is > configured, it will kill the newest running or suspended process. If a > little bit is configured, it will probably kill some idle process, like > zfsd, that is swapped out because it doesn't run very often. > > * Even if multiple GB of swap are configured, the OOM killer is still > biased towards killing idle processes when shortage == VM_OOM_SWAPZ. Most > often, the process responsible for an out-of-memory condition is not idle, > and is consuming large amounts of RAM. > > * It ignores RLIMIT_RSS. We consider that rlimit when deciding whether to > move a process from RAM to swap. > > * The "out of swap space" kernel message doesn't specify whether the > process was killed because of insufficient swap or RAM (the shortage > variable) > > I propose the following changes: > > * Incorporate shortage into the "out of swap space" message. ok with me, not sure if users could make any action based on discretion > * When walking the process list, if any process exceeds its RLIMIT_RSS, > choose it immediately, without bothering to compare it to older processes. RSS was never supposed to be a limit on how many pages are resident. It only provided some preference for more aggressive paging out process' pages. Or put it differently, RSS is not supposed to be the working set size in VMS/NT sense. > * Always consider the sum of a process's RAM + swap, regardless of the > shortage variable. > > Does this make sense? Am I missing something about shortage == > VM_OOM_SWAPZ? I don't understand why you would ever want to exclude > processes' RAM usage. That logic was added in revision > 2025d69ba7a68a5af173007a8072c45ad797ea23, but I don't understand the > rationale. SWAPZ means that swap zone is exhausted. In this case, killing a process that does not use swap, would not free any space in the zone. Similarly, we should select a process with largest swap (== metadata kept in swap zone) use to free something in swap zone. In other words, such kill could be not enough and really require more and more rounds of OOM, esp. on machine with very small swap configured. From owner-freebsd-hackers@freebsd.org Tue Feb 23 21:20:34 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6F7A25533AB for ; Tue, 23 Feb 2021 21:20:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlX5Z2hRJz4f1G for ; Tue, 23 Feb 2021 21:20:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oo1-f43.google.com with SMTP id f26so4227838oog.5 for ; Tue, 23 Feb 2021 13:20:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PxltA2VW4YcjXh7HVedjX5XRSW8KohZcPV3CUciH1jk=; b=s3G7ghg697Frw5jRbzl3P70EwPAbbZh+PY6flGZPSN4G/7Ioaof7a5+vCkhAPpxtVi iBFrI7Mts+luTXFsmL8zUNe4b9PgsutAY+EIwCq1Nyzv0jdY7G2WXJL07CcmtDWDeFvP JSIZ7PSEmcUjxvDKgbYEpnNuarGqKxncGfntjh4eW7jOx+BMlh3o1lv3Jtu2hkN4KuhL Rr3aq+cbbrq1HfSN2FO1RyTQna71uPUH5+gur6vbeAnmpBlozfqXx4QkPRcXBIJg1/cr 7atnW8CI91Sfi6rt5Upp4iNgJqnga3nMm35N/6DkSYr6S9HjVTVoNZUPumxCeuT2Exqc kZTw== X-Gm-Message-State: AOAM531MzQbLwUBbtBCs7OATliV1uNqoCQ0tpRlw9obrs7Gn9U3fcffy ckVgsKD3YB3poShprOb8bygR0TOg9mH8RzxQi+qrDxYiV+0= X-Google-Smtp-Source: ABdhPJxcisB2buG6fehMGQOpQB9LcPxelee54CPKLZPpaCOr/FA40Cpt5dIPBZPgDhXiJbeZBslljeAGxsg12H0d/X0= X-Received: by 2002:a4a:e9a6:: with SMTP id t6mr19758623ood.74.1614115233154; Tue, 23 Feb 2021 13:20:33 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 23 Feb 2021 14:20:21 -0700 Message-ID: Subject: Re: The out-of-swap killer makes poor choices To: Konstantin Belousov Cc: FreeBSD Hackers X-Rspamd-Queue-Id: 4DlX5Z2hRJz4f1G X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 21:20:34 -0000 On Tue, Feb 23, 2021 at 2:11 PM Konstantin Belousov wrote: > On Tue, Feb 23, 2021 at 01:49:49PM -0700, Alan Somers wrote: > > To me it's always seemed like the out-of-swap killer kills the wrong > > process. Oh, it does the right thing with a trivial while(1) {malloc()} > > test program, but not with real workloads. To summarize the logic in > > vm_pageout_oom: > > > > * Don't kill system, protected, or killed processes > > * Don't kill processes with a thread that isn't running or suspended > > * Kill whichever process is using the most swap or swap + ram, depending > on > > the shortage variable. On ties, kill the newest one. > > > > This algorithm probably made sense in the days when computers had much > more > > swap than RAM. But now it leads to several problems: > > > > * It's almost guaranteed to do the wrong thing when shortage == > > VM_OOM_SWAPZ and there is little or no swap configured. If no swap is > > configured, it will kill the newest running or suspended process. If a > > little bit is configured, it will probably kill some idle process, like > > zfsd, that is swapped out because it doesn't run very often. > > > > * Even if multiple GB of swap are configured, the OOM killer is still > > biased towards killing idle processes when shortage == VM_OOM_SWAPZ. > Most > > often, the process responsible for an out-of-memory condition is not > idle, > > and is consuming large amounts of RAM. > > > > * It ignores RLIMIT_RSS. We consider that rlimit when deciding whether > to > > move a process from RAM to swap. > > > > * The "out of swap space" kernel message doesn't specify whether the > > process was killed because of insufficient swap or RAM (the shortage > > variable) > > > > I propose the following changes: > > > > * Incorporate shortage into the "out of swap space" message. > ok with me, not sure if users could make any action based on discretion > > > * When walking the process list, if any process exceeds its RLIMIT_RSS, > > choose it immediately, without bothering to compare it to older > processes. > RSS was never supposed to be a limit on how many pages are resident. > It only provided some preference for more aggressive paging out process' > pages. > > Or put it differently, RSS is not supposed to be the working set size > in VMS/NT sense. > Sure, but given that we must kill _something_, preferentially killing a process that was specifically limited sounds better than killing a process that wasn't, won't you agree? > > > * Always consider the sum of a process's RAM + swap, regardless of the > > shortage variable. > > > > Does this make sense? Am I missing something about shortage == > > VM_OOM_SWAPZ? I don't understand why you would ever want to exclude > > processes' RAM usage. That logic was added in revision > > 2025d69ba7a68a5af173007a8072c45ad797ea23, but I don't understand the > > rationale. > > SWAPZ means that swap zone is exhausted. In this case, killing a process > that does not use swap, would not free any space in the zone. Similarly, > we should select a process with largest swap (== metadata kept in swap > zone) > use to free something in swap zone. > But killing a process that does not use swap could reduce the need for more swap by other processes. How many cases are there where a process needs more SWAP and won't settle for RAM instead? > > In other words, such kill could be not enough and really require more and > more rounds of OOM, esp. on machine with very small swap configured. > From owner-freebsd-hackers@freebsd.org Tue Feb 23 21:24:04 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 576435537D5 for ; Tue, 23 Feb 2021 21:24:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.gq1.yahoo.com (sonic311-24.consmr.mail.gq1.yahoo.com [98.137.65.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlX9b3KfLz4fZ7 for ; Tue, 23 Feb 2021 21:24:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1614115441; bh=Hi/dq99XsMym3XrxRi8QGkjBrBb2sdAN1p5lPO5lAOu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=AY8xDKSrG+pnDXFW3EYawH+s9LpnNhQ16p9i2GZYRcCReuSIMsr3aKGNvtpD41vogZCx3zW+JtfMGHCSx0ZsionRnCD0T373OSZFABcy0ry/ZC7jS7rTglzeqezCRQyRRspYGPJct7IE1XxP2FovHS6qqnB3Ze7ErBCSVxqhMY9xFv2lFiTkIOb2L3hJiZNTP8pxcZwxX1+XwIEmwQvEa7t8q5kb2AIqVVuNJfdaLEGRKOc6gXjdMiUhr+HdeBdF+OJFxcKnAG99CG0UOW+gHqr7Dtk9RZCNrvCwG/9+YxzVXPpLtDTAI/HJm49tF/Rnir78X+hF085dDaNfzA2tiw== X-YMail-OSG: o0RRbo8VM1nJbY7A_QjhUxZ73HDed7h96uwr32yfe7PgygJ3ZvWv_nSzl57JdT0 Ydn6WcohN6bYPtEtx3NlGxSQIuq5axy52nFi3skq.0G1EJqyecIrKBl5SoBnnXPCQowhCrZLxpeH wzvgVvdFr_mTodXJx5vO2Rtu9iNwyhUcxUX498nl0XJucS324ff9mE3Qiad1xj0UXX7PEwxhxuA8 9dChBptP0._PDHS9MnzBgHNBi_J.GUrMeWvwNWkwMwWHaJH7tz9gEgJ9FYqKg05WvtSIMkOdL4cD oaMFkjVJHVP6XD6NTrb45iwBx11jM1wr0oBqRDx0IU70hvpvagpWdhYroROzwoUGV2ajio3eEZzz 1dUdfhIcw03CV8eBa6Q1h_CHEVHdrPBAyv99ZYvKUVcYmzVJqn0m.OVtORGUf5wAMcNYXE6CsJLU 3zvQxkF03Hw6s45x4M7kBH4Wx8cGj6so.jBZRFKq4XqoluyX0JOkGJ1winU7yLxY5uJ51PoXn8za hk6EP7CJnWYSefqvKzLnG0lU2ZWVwOlGGOjwjn7fyLEP0ObpJaf.yYPSihAM6QU2sHYyhTHaTJyu qPHFbIg97D_ZVhQXi9rpVWqBFgfSOfSI4.g4QdAlzljdUVC309fNlcrmC7HSeoTFMarhzhUW2.QD vlcLa81.U0r2ob13Q2jUQJb3jWV1sKD.dF.KEmL7M_r8Z1kuFr0gIjrO4s4W0joxSjCiGp3uDzJw 9hMz8PrZo5YBC3U77ZEF_bI2.QFNVlcFdKisc96NFzhegG7rnZwDPh._S9E6HHMzb3sqbTHI9rlo n1nfGaMYQhLU2MVTCI4uCoMxK5D0qjoFAeDqSB1sBNZ4ijGNuJjhRvPsYTJto5xZent3odxha.8V HLZkdozuYVMx2F4XDmQ1lqFRKFDu0xiGpje5CTUE.uXh4Me7OtcIczrUkCnEEWNPNlX0oFiMEAYj e.Q1HTLCNLmUuZKTF20e39QXtiynOO7DQgFk1lFacmBauv8rZb8sjk7YRmtnhhN2vECZtmx_kT53 xu2.Wdxg.rTSukqlXZmyFH3B15cXgVC_RKC87b0zVIKSEwlIJCXgo3rkgA6cKcIvdiyGpneE1ui0 qYuivYXvTlVWWyCznQOcc0gNPNu8xpXyCOfpCF0QXrwxkfC7llzkUmCSUNaRCaOACgBQLzDpBQy0 vuxTYTFpD5j1FvEgodOQvyY024_Y_CRK_yLqfPsCbASjRcGxaxnavU20Rw387s86ahCxiHzAw1mV CRM.yqD_2gt0dMKut1D3SY5zbF_SLm0B1_AVQHwa_QIBzECozNv5ZEftltsEHEVXOdLITv7kah2X k_5Q7o6MvFbSY53T4VPADx6.WpdTvaVFDOQ1TAk3RvVtN9HmOE.7T1APENCH7T75hoR3ZwXIfVcB .lrV1cK65O9MvIaYjZP7IGHb8kR3tkz0cNQXPuFLygt9AnQEVCAtlpJzvW8W2iru94Jv4Q0z9NE4 aN.Pq_8UXtBVxTLNwxGX.q9fitdErh4g.HN5dGTfCSnj0FD5sKtWDNOR1gGXo8RcQhuRx4E9a9Yd 1TMeA4sCr0wX3hWLmhHKbimDwUPtNL.ZioZ5Yt8WYrcoV7wHnd5qVhhgnKRgT6xy0QCCpt9fFLiD nYE1lx7d.XVOrb6oErxXo0qf4ik11VMNr5f4mw9QLZK5lbdeVigfn9OtsninQb7q2ZMOEfEay_nL yxPOIGTXSATDwgUkAcnDpFMd_mGJqzFRU3f.T09feZ9xgfujpQfXfW2UpaAOOWNL5LrCzW3I8QOw 2HKSh6kJPZ1CvMCnp7H87oPc69k4xZt4RV6mV75CPACwNpbK6CV9LTQSJUfZGsDyAd8MBnbAE7N5 0wciuV6zSUBq5oM3gu0uKKLiy9tOBrsMZUqwHAG3S1Jt2uTYvZQcYSWNgYcNg5l1HBHfiM.x6HY9 t3.lZ1ri9ZmzTNmwbfK7kZv9lYRkOoMk1Qydx8aNvM15eQNl9EeDe2lcCBxRHJ61w8cnDHsCOPJx ErQurq31bfFFKQUtggrkkvsNe5gg.3bbiaQV4O3tz9svbHwoKdhGTKd4u_45dbBbJte6WHDsXWUH Vg.CuYnL_nxUUrb8vFW_to3H6lwR02k13NueYFe1cptZEnP1aVSdzK2qjoYcv2eDqLlM6PWAAZbI m5Db_ALUlMNlllTrMjiROOU_9rxBu4RGhX8gfkgTsUzpSIMC2DDPxIrYXZJ.Q8QjxDUP5GfCxWt4 FDENozYY6M92OvK3UB965H8cEZVrU4L9j0Q_taVAKCrkR0Sgv6vZNM6B41WYp2.u3txK3llDjFdB zGi0ctS1hp2a90aXFVV3rF4rei6CGwXe4SEiALGIKuCVaHS._98tZ7vp.TePlk.2Ra..UE0u7vo1 4fYrThVv7NwUvcR6kt0TsnRDyn5c0r4nfElfB9hxBsQsWZU45krL9KjW.e0Mpp4SXwJIRgTkCfGg xM98gnpXLwtxD9jYSwST2PQPHTC_6ldLbm0wnK7hmnkhHWp4dEMxlMQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Tue, 23 Feb 2021 21:24:01 +0000 Received: by smtp423.mail.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 3aa986d8c7310bca367b3d92bd394e1f; Tue, 23 Feb 2021 21:23:57 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: The out-of-swap killer makes poor choices Message-Id: <93DA798A-1109-48B0-AD5E-063B5A182BFB@yahoo.com> Date: Tue, 23 Feb 2021 13:23:56 -0800 To: asomers@freebsd.org, FreeBSD Hackers X-Mailer: Apple Mail (2.3654.60.0.2.21) References: <93DA798A-1109-48B0-AD5E-063B5A182BFB.ref@yahoo.com> X-Rspamd-Queue-Id: 4DlX9b3KfLz4fZ7 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.65.205:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.65.205:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.205:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.205:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 21:24:04 -0000 Alan Somers asomers at freebsd.org wrote on Tue Feb 23 20:50:02 UTC 2021 : . . . > * The "out of swap space" kernel message doesn't specify whether the > process was killed because of insufficient swap or RAM (the shortage > variable) . . . I'm only dealing with "why" notifications part of the Email. I'm not sure your notes are complete for coverage, although I can not claim to fully understand the implications of the below. So, just some things to consider in that area . . . When I looked at the code I found 4 things that lead to the same "out of swap space" messages for which no "swap_pager_getswapspace(...): failed" seemed to need to be involved: Sustained low free RAM (via stays-runnable processes). A sufficiently delayed pageout. The swap blk uma zone was exhausted. The swap pctrie uma zone was exhausted. (I run a modified kernel that reports messages about which of the 4 initiated the OOM. I depend on the "swap_pager_getswapspace(...): failed" notices to detect actual out of swap/paging space conditions.) The first 2 of the 4 above have some tunables: # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes. The # delay is a count of kernel-attempts to gain # free RAM (so not time units). vm.pageout_oom_seq=3D120 (The default is 12 as far as I know. I systematically use the above value.) NOTE: stable/12 -r351776 got the support for the following: (I've not checked the match to releases.) # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=3D-1 (I systematically use the above value but am careful to strongly expect that I'd not actually run out of swap space.) That last has the alternative structure needed for when out of swap is a concern as I understand what I've been told/read (replace ???'s with notation for positive integers): # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes: #vm.pfault_oom_attempts=3D ??? #vm.pfault_oom_wait=3D ??? # (The multiplication of the two values is the # total but there are other potential tradoffs # in the factors multiplied for the same total.) For reference: # sysctl -d vm.pfault_oom_wait vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler # sysctl -d vm.pfault_oom_attempts vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling # sysctl -d vm.pageout_oom_seq vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM I'm definitely not a fan of the misleading "out of swap space" notices. They greatly mislead me until Mark J. got involved and did some basic fixing to my understanding of the context, including pointing out vm.pageout_oom_seq at the time. (vm.pfault_oom_attempts and vm.pfault_oom_wait are from a later discovery.) It seems that the detailed reason for the OOM helps drive the appropriate future configuration choices for avoiding or managing things. In my view the the specific reason should be explicitly reported. If nothing else, it provides context to indicate in a question to the lists about what is then appropriate to do give the occurrence observed. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Tue Feb 23 22:36:31 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C59DD5554DA for ; Tue, 23 Feb 2021 22:36:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlYnB6qQKz4lPS; Tue, 23 Feb 2021 22:36:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11NMaL9N048759 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 24 Feb 2021 00:36:25 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11NMaL9N048759 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11NMaLRH048758; Wed, 24 Feb 2021 00:36:21 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 24 Feb 2021 00:36:21 +0200 From: Konstantin Belousov To: Alan Somers Cc: FreeBSD Hackers Subject: Re: The out-of-swap killer makes poor choices Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DlYnB6qQKz4lPS X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [0.14 / 15.00]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_SPAM_MEDIUM(0.16)[0.157]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_SHORT(0.98)[0.980]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 22:36:31 -0000 On Tue, Feb 23, 2021 at 02:20:21PM -0700, Alan Somers wrote: > On Tue, Feb 23, 2021 at 2:11 PM Konstantin Belousov > wrote: > > > On Tue, Feb 23, 2021 at 01:49:49PM -0700, Alan Somers wrote: > > > To me it's always seemed like the out-of-swap killer kills the wrong > > > process. Oh, it does the right thing with a trivial while(1) {malloc()} > > > test program, but not with real workloads. To summarize the logic in > > > vm_pageout_oom: > > > > > > * Don't kill system, protected, or killed processes > > > * Don't kill processes with a thread that isn't running or suspended > > > * Kill whichever process is using the most swap or swap + ram, depending > > on > > > the shortage variable. On ties, kill the newest one. > > > > > > This algorithm probably made sense in the days when computers had much > > more > > > swap than RAM. But now it leads to several problems: > > > > > > * It's almost guaranteed to do the wrong thing when shortage == > > > VM_OOM_SWAPZ and there is little or no swap configured. If no swap is > > > configured, it will kill the newest running or suspended process. If a > > > little bit is configured, it will probably kill some idle process, like > > > zfsd, that is swapped out because it doesn't run very often. > > > > > > * Even if multiple GB of swap are configured, the OOM killer is still > > > biased towards killing idle processes when shortage == VM_OOM_SWAPZ. > > Most > > > often, the process responsible for an out-of-memory condition is not > > idle, > > > and is consuming large amounts of RAM. > > > > > > * It ignores RLIMIT_RSS. We consider that rlimit when deciding whether > > to > > > move a process from RAM to swap. > > > > > > * The "out of swap space" kernel message doesn't specify whether the > > > process was killed because of insufficient swap or RAM (the shortage > > > variable) > > > > > > I propose the following changes: > > > > > > * Incorporate shortage into the "out of swap space" message. > > ok with me, not sure if users could make any action based on discretion > > > > > * When walking the process list, if any process exceeds its RLIMIT_RSS, > > > choose it immediately, without bothering to compare it to older > > processes. > > RSS was never supposed to be a limit on how many pages are resident. > > It only provided some preference for more aggressive paging out process' > > pages. > > > > Or put it differently, RSS is not supposed to be the working set size > > in VMS/NT sense. > > > > Sure, but given that we must kill _something_, preferentially killing a > process that was specifically limited sounds better than killing a process > that wasn't, won't you agree? Semantic of RLIMIT_RSS is not to limit, but to give preference for pageout. Changing it to the semantic of 'preference for OOM' would give the similar complaint. > > > > > > > * Always consider the sum of a process's RAM + swap, regardless of the > > > shortage variable. > > > > > > Does this make sense? Am I missing something about shortage == > > > VM_OOM_SWAPZ? I don't understand why you would ever want to exclude > > > processes' RAM usage. That logic was added in revision > > > 2025d69ba7a68a5af173007a8072c45ad797ea23, but I don't understand the > > > rationale. > > > > SWAPZ means that swap zone is exhausted. In this case, killing a process > > that does not use swap, would not free any space in the zone. Similarly, > > we should select a process with largest swap (== metadata kept in swap > > zone) > > use to free something in swap zone. > > > > But killing a process that does not use swap could reduce the need for more > swap by other processes. How many cases are there where a process needs > more SWAP and won't settle for RAM instead? Both choices are somewhat random. The goal is to get more swap zone slack, and this is what the code tried to target. In fact, if OOM kills largest RAM+swap consumer, then with the small swap there is huge chance that swap is not freed, and then on the next nearby pageout attempt some more process would be killed, perhaps innocently. OOM purpose is not to smoother operation of over-committed system, but to have it survive (avoid low resources deadlock) to the state where it can be examined and possibly corrected. > > > > > > In other words, such kill could be not enough and really require more and > > more rounds of OOM, esp. on machine with very small swap configured. > > From owner-freebsd-hackers@freebsd.org Tue Feb 23 23:29:58 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8C9F4557F38 for ; Tue, 23 Feb 2021 23:29:58 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlZyt3QYVz4qNT for ; Tue, 23 Feb 2021 23:29:58 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f171.google.com with SMTP id q186so476470oig.12 for ; Tue, 23 Feb 2021 15:29:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/ONPizlIIQkiiFQ/xGopnPxsmsMe4f43nFk5XDRBFc8=; b=RQtikmPKxsoI8KAbMHh8G7HAI83EmK8C7eD7j2t/vFPTmGEeDn5Ib/6w10XQrp6JBv 8cClKvEIXqg8XxdQ+xLn+Cb45EtkYljVxJKUumsClD2GdnCx+hRSWQ7P0LzNrnWvCpJH NvcSamJ+w5RxH0NZIbOKnrkRt+3X2xxoADJ5QxMnuJ/T8fzzaHuEkXvFh8paUCNIdRJ5 dsS0y1N08lWo21qtiPjxElGXzr8OvKO2NMS/gZbKATyPJ6C8CLYeU1iXvK63lqZ4iepB QIhD+1IfDk8zAYhg+cGlAUb/UktffDj/Y1uWvXIIXxOrHK6hpPh6kxjS1CYdY7pCaoIi 2mSw== X-Gm-Message-State: AOAM532753l9YlCVFgmcfNug9s8ThZAOX4N1WyvQfZIkcmPv2mxY7acf 8eYzHikY1dq6G/rGarnfuSzC/M9uCWhRdNHIewI= X-Google-Smtp-Source: ABdhPJzR2nKBDr3UgOwZ3umVeh4XPFnZ0gUURz9FzTLtASqQx9YJRKMNkfDLsStSll/MSD8ih3KgdNTcpr0GqggaYS8= X-Received: by 2002:aca:478a:: with SMTP id u132mr825333oia.73.1614122997276; Tue, 23 Feb 2021 15:29:57 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 23 Feb 2021 16:29:46 -0700 Message-ID: Subject: Re: The out-of-swap killer makes poor choices To: Konstantin Belousov Cc: FreeBSD Hackers X-Rspamd-Queue-Id: 4DlZyt3QYVz4qNT X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 23:29:58 -0000 On Tue, Feb 23, 2021 at 3:36 PM Konstantin Belousov wrote: > On Tue, Feb 23, 2021 at 02:20:21PM -0700, Alan Somers wrote: > > On Tue, Feb 23, 2021 at 2:11 PM Konstantin Belousov > > > wrote: > > > > > On Tue, Feb 23, 2021 at 01:49:49PM -0700, Alan Somers wrote: > > > > To me it's always seemed like the out-of-swap killer kills the wrong > > > > process. Oh, it does the right thing with a trivial while(1) > {malloc()} > > > > test program, but not with real workloads. To summarize the logic in > > > > vm_pageout_oom: > > > > > > > > * Don't kill system, protected, or killed processes > > > > * Don't kill processes with a thread that isn't running or suspended > > > > * Kill whichever process is using the most swap or swap + ram, > depending > > > on > > > > the shortage variable. On ties, kill the newest one. > > > > > > > > This algorithm probably made sense in the days when computers had > much > > > more > > > > swap than RAM. But now it leads to several problems: > > > > > > > > * It's almost guaranteed to do the wrong thing when shortage == > > > > VM_OOM_SWAPZ and there is little or no swap configured. If no swap > is > > > > configured, it will kill the newest running or suspended process. > If a > > > > little bit is configured, it will probably kill some idle process, > like > > > > zfsd, that is swapped out because it doesn't run very often. > > > > > > > > * Even if multiple GB of swap are configured, the OOM killer is still > > > > biased towards killing idle processes when shortage == VM_OOM_SWAPZ. > > > Most > > > > often, the process responsible for an out-of-memory condition is not > > > idle, > > > > and is consuming large amounts of RAM. > > > > > > > > * It ignores RLIMIT_RSS. We consider that rlimit when deciding > whether > > > to > > > > move a process from RAM to swap. > > > > > > > > * The "out of swap space" kernel message doesn't specify whether the > > > > process was killed because of insufficient swap or RAM (the shortage > > > > variable) > > > > > > > > I propose the following changes: > > > > > > > > * Incorporate shortage into the "out of swap space" message. > > > ok with me, not sure if users could make any action based on discretion > > > > > > > * When walking the process list, if any process exceeds its > RLIMIT_RSS, > > > > choose it immediately, without bothering to compare it to older > > > processes. > > > RSS was never supposed to be a limit on how many pages are resident. > > > It only provided some preference for more aggressive paging out > process' > > > pages. > > > > > > Or put it differently, RSS is not supposed to be the working set size > > > in VMS/NT sense. > > > > > > > Sure, but given that we must kill _something_, preferentially killing a > > process that was specifically limited sounds better than killing a > process > > that wasn't, won't you agree? > Semantic of RLIMIT_RSS is not to limit, but to give preference for pageout. > Changing it to the semantic of 'preference for OOM' would give the similar > complaint. > > > > > > > > > > > > * Always consider the sum of a process's RAM + swap, regardless of > the > > > > shortage variable. > > > > > > > > Does this make sense? Am I missing something about shortage == > > > > VM_OOM_SWAPZ? I don't understand why you would ever want to exclude > > > > processes' RAM usage. That logic was added in revision > > > > 2025d69ba7a68a5af173007a8072c45ad797ea23, but I don't understand the > > > > rationale. > > > > > > SWAPZ means that swap zone is exhausted. In this case, killing a > process > > > that does not use swap, would not free any space in the zone. > Similarly, > > > we should select a process with largest swap (== metadata kept in swap > > > zone) > > > use to free something in swap zone. > > > > > > > But killing a process that does not use swap could reduce the need for > more > > swap by other processes. How many cases are there where a process needs > > more SWAP and won't settle for RAM instead? > Both choices are somewhat random. The goal is to get more swap zone slack, > and this is what the code tried to target. > > In fact, if OOM kills largest RAM+swap consumer, then with the small swap > there is huge chance that swap is not freed, and then on the next nearby > pageout attempt some more process would be killed, perhaps innocently. > > OOM purpose is not to smoother operation of over-committed system, but > to have it survive (avoid low resources deadlock) to the state where it > can be examined and possibly corrected. > > > > > > > > > > > In other words, such kill could be not enough and really require more > and > > > more rounds of OOM, esp. on machine with very small swap configured. > > Ok, I'll abandon this idea. From owner-freebsd-hackers@freebsd.org Wed Feb 24 11:02:38 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5A8A755867A for ; Wed, 24 Feb 2021 11:02:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DltL50xWdz4qDF; Wed, 24 Feb 2021 11:02:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11OB2RTJ028201 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 24 Feb 2021 13:02:30 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11OB2RTJ028201 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11OB2R3N028200; Wed, 24 Feb 2021 13:02:27 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 24 Feb 2021 13:02:27 +0200 From: Konstantin Belousov To: Alan Somers Cc: FreeBSD Hackers Subject: Re: The out-of-swap killer makes poor choices Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DltL50xWdz4qDF X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-1.04 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.996]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_SPAM_SHORT(0.95)[0.953]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 11:02:38 -0000 On Tue, Feb 23, 2021 at 04:29:46PM -0700, Alan Somers wrote: > On Tue, Feb 23, 2021 at 3:36 PM Konstantin Belousov > wrote: > > > On Tue, Feb 23, 2021 at 02:20:21PM -0700, Alan Somers wrote: > > > On Tue, Feb 23, 2021 at 2:11 PM Konstantin Belousov > > > > > wrote: > > > > > > > On Tue, Feb 23, 2021 at 01:49:49PM -0700, Alan Somers wrote: > > > > > To me it's always seemed like the out-of-swap killer kills the wrong > > > > > process. Oh, it does the right thing with a trivial while(1) > > {malloc()} > > > > > test program, but not with real workloads. To summarize the logic in > > > > > vm_pageout_oom: > > > > > > > > > > * Don't kill system, protected, or killed processes > > > > > * Don't kill processes with a thread that isn't running or suspended > > > > > * Kill whichever process is using the most swap or swap + ram, > > depending > > > > on > > > > > the shortage variable. On ties, kill the newest one. > > > > > > > > > > This algorithm probably made sense in the days when computers had > > much > > > > more > > > > > swap than RAM. But now it leads to several problems: > > > > > > > > > > * It's almost guaranteed to do the wrong thing when shortage == > > > > > VM_OOM_SWAPZ and there is little or no swap configured. If no swap > > is > > > > > configured, it will kill the newest running or suspended process. > > If a > > > > > little bit is configured, it will probably kill some idle process, > > like > > > > > zfsd, that is swapped out because it doesn't run very often. > > > > > > > > > > * Even if multiple GB of swap are configured, the OOM killer is still > > > > > biased towards killing idle processes when shortage == VM_OOM_SWAPZ. > > > > Most > > > > > often, the process responsible for an out-of-memory condition is not > > > > idle, > > > > > and is consuming large amounts of RAM. > > > > > > > > > > * It ignores RLIMIT_RSS. We consider that rlimit when deciding > > whether > > > > to > > > > > move a process from RAM to swap. > > > > > > > > > > * The "out of swap space" kernel message doesn't specify whether the > > > > > process was killed because of insufficient swap or RAM (the shortage > > > > > variable) > > > > > > > > > > I propose the following changes: > > > > > > > > > > * Incorporate shortage into the "out of swap space" message. > > > > ok with me, not sure if users could make any action based on discretion > > > > > > > > > * When walking the process list, if any process exceeds its > > RLIMIT_RSS, > > > > > choose it immediately, without bothering to compare it to older > > > > processes. > > > > RSS was never supposed to be a limit on how many pages are resident. > > > > It only provided some preference for more aggressive paging out > > process' > > > > pages. > > > > > > > > Or put it differently, RSS is not supposed to be the working set size > > > > in VMS/NT sense. > > > > > > > > > > Sure, but given that we must kill _something_, preferentially killing a > > > process that was specifically limited sounds better than killing a > > process > > > that wasn't, won't you agree? > > Semantic of RLIMIT_RSS is not to limit, but to give preference for pageout. > > Changing it to the semantic of 'preference for OOM' would give the similar > > complaint. > > > > > > > > > > > > > > > > > * Always consider the sum of a process's RAM + swap, regardless of > > the > > > > > shortage variable. > > > > > > > > > > Does this make sense? Am I missing something about shortage == > > > > > VM_OOM_SWAPZ? I don't understand why you would ever want to exclude > > > > > processes' RAM usage. That logic was added in revision > > > > > 2025d69ba7a68a5af173007a8072c45ad797ea23, but I don't understand the > > > > > rationale. > > > > > > > > SWAPZ means that swap zone is exhausted. In this case, killing a > > process > > > > that does not use swap, would not free any space in the zone. > > Similarly, > > > > we should select a process with largest swap (== metadata kept in swap > > > > zone) > > > > use to free something in swap zone. > > > > > > > > > > But killing a process that does not use swap could reduce the need for > > more > > > swap by other processes. How many cases are there where a process needs > > > more SWAP and won't settle for RAM instead? > > Both choices are somewhat random. The goal is to get more swap zone slack, > > and this is what the code tried to target. > > > > In fact, if OOM kills largest RAM+swap consumer, then with the small swap > > there is huge chance that swap is not freed, and then on the next nearby > > pageout attempt some more process would be killed, perhaps innocently. > > > > OOM purpose is not to smoother operation of over-committed system, but > > to have it survive (avoid low resources deadlock) to the state where it > > can be examined and possibly corrected. > > > > > > > > > > > > > > > > In other words, such kill could be not enough and really require more > > and > > > > more rounds of OOM, esp. on machine with very small swap configured. > > > > > Ok, I'll abandon this idea. No OOM algorithm would ever satisfy everybody. I explained the reasoning for the current design, even if it actually evolved this way, instead being written as a whole with the stated goal. I do not object against adding something that would help to get it more fit with different goals as well, but the current idea of making the system survive should be kept. I remember Linux has more advanced controls to guide OOM decisions. We only have 'protected' flag that should prevent killer from ever touching specific process, like sshd. From owner-freebsd-hackers@freebsd.org Wed Feb 24 12:02:25 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF0FA55A9B8 for ; Wed, 24 Feb 2021 12:02:25 +0000 (UTC) (envelope-from olivier.freebsd@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [IPv6:2a01:e0c:1:1599::15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dlvg561Jfz4tQr; Wed, 24 Feb 2021 12:02:25 +0000 (UTC) (envelope-from olivier.freebsd@free.fr) Received: from ravel.localnet (unknown [90.118.181.206]) (Authenticated sender: olivier.freebsd@free.fr) by smtp6-g21.free.fr (Postfix) with ESMTPSA id 610BC78035B; Wed, 24 Feb 2021 13:02:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=free.fr; s=smtp-20201208; t=1614168142; bh=/6D6zmMQDU0iWKWYt1BoHN3XYPIV3RLYvs8uPbK2TpI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fhEm8Ee+GMXlUPrM+XnSItCyek6mOynkD89vwS1e0s674TrOtzXeyznzI7A4kakaT B3hJIaVrj7a3pSPxcQ4vIA6YLONiGHs0PYs6+QBZxaySbky3bSOt0AYM4dOGTxv8w1 e0XtPnfccrYffYv4q+dXJUkS7ZcajZDlsgFMo8x+K9uXTuXclw+0FCkwLbgLJTNSg2 qWo4HKXi6Su4fyhG3KoPqR4wKmr/nZ14aYeAmDIlkNlDUFy1AjDFMD5bIlrc4Vxxek 2B0lqmj2yoLG2cCznT9RTbBVfdb/Fj9HhsdxSazHCQ99AI0vsuESFYgZWrtX5iuHXt xzoTCGHGb/L9Q== From: Olivier Certner To: Alan Somers , Konstantin Belousov Cc: FreeBSD Hackers Subject: Re: The out-of-swap killer makes poor choices Date: Wed, 24 Feb 2021 13:02:14 +0100 Message-ID: <1984125.0OzZcVfBr4@ravel> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Dlvg561Jfz4tQr X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 12:02:26 -0000 Hi, > > Ok, I'll abandon this idea. I hope you don't abandon the idea of improving the OOM killer in the long term if you feel that something is wrong. > I explained the reasoning for the current design, even if it actually > evolved this way, instead being written as a whole with the stated goal. > I do not object against adding something that would help to get it more > fit with different goals as well, but the current idea of making the > system survive should be kept. So true. The main goal (system survival) does not prevent (not so) secondary ones. I'm sorry not to have any technical contribution to propose, but instead I have some testimony that may interest you, although old. 2 to 3 years ago, I stumbled against production problems on servers doing heavy computations. Only a few processes (2 generally) were doing them, and most of the time consumed less than 1/4 of the available RAM (2 GiB). Apart from that, no other process was allocating any significant amount of memory. Only some base default daemons (syslogd, cron) and sshd were running. Occasionally, very big jobs would come, and one or more of these processes would start eating up all available memory, until FreeBSD decided that it was time to take action. Sometimes it would decide to kill one of these processes, but more often than not sshd or cron were killed instead, although they were consuming ridiculous amounts of memory. I tried tweaks via vm.pageout_oom_seq (I think I set it to 120, as Mark did) and vm.pfault_oom_attempts, without much change. In the end, I decided to use 'protect', via rc.conf's '*_oomprotect="YES"' facility, to workaround this problem and save me some headaches. At some point, some of these machines had swap configured (separate AWS disks), but later I removed swap entirely. What I report occurred for the latter configuration, but IIRC I observed similar behavior in the former. I have not had this use case since then, so I can't say if this has been fixed (by commits such as r353734/d307bdcc2c473858) or not. -- Olivier Certner From owner-freebsd-hackers@freebsd.org Wed Feb 24 12:38:33 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BF2BB55BF8A for ; Wed, 24 Feb 2021 12:38:33 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from edna.lautre.net (edna.lautre.net [80.67.160.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "lautre.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DlwSm6lVqz3C7y for ; Wed, 24 Feb 2021 12:38:32 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from graf.pompo.net (graf.pompo.net [78.225.128.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by edna.lautre.net (Postfix) with ESMTPSA id 7510D11D956 for ; Wed, 24 Feb 2021 13:38:21 +0100 (CET) Received: by graf.pompo.net (Postfix, from userid 1001) id B05417C7C56; Wed, 24 Feb 2021 13:38:17 +0100 (CET) Date: Wed, 24 Feb 2021 13:38:17 +0100 From: Thierry Thomas To: freebsd-hackers@freebsd.org Subject: Re: The out-of-swap killer makes poor choices Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <1984125.0OzZcVfBr4@ravel> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7RIpcGXDOQOnB0f9" Content-Disposition: inline In-Reply-To: <1984125.0OzZcVfBr4@ravel> X-Operating-System: FreeBSD 12.2-STABLE amd64 Organization: Kabbale Eros X-Face: (hRbQnK~Pt7$ct`!fupO(`y_WL4^-Iwn4@ly-.,[4xC4xc; y=\ipKMNm<1J>lv@PP~7Z<.t KjAnXLs: X-PGP: 0xF1C516B3C8359753 X-Rspamd-Queue-Id: 4DlwSm6lVqz3C7y X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of thierry@pompo.net designates 80.67.160.88 as permitted sender) smtp.mailfrom=thierry@pompo.net X-Spamd-Result: default: False [-5.07 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.975]; SIGNED_PGP(-2.00)[]; FORGED_SENDER(0.30)[thierry@freebsd.org,thierry@pompo.net]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[80.67.160.88:from]; ASN(0.00)[asn:20766, ipnet:80.67.160.0/19, country:FR]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[thierry@freebsd.org,thierry@pompo.net]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[thierry]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[80.67.160.88:from:127.0.2.255]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 12:38:33 -0000 --7RIpcGXDOQOnB0f9 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Le mer. 24 f=E9vr. 21 =E0 13:02:14 +0100, Olivier Certner =E9crivait=A0: > Hi, Hello, > 2 to 3 years ago, I stumbled against production problems on servers doing= =20 > heavy computations. Only a few processes (2 generally) were doing them, a= nd=20 > most of the time consumed less than 1/4 of the available RAM (2 GiB). Apa= rt=20 > from that, no other process was allocating any significant amount of memo= ry.=20 > Only some base default daemons (syslogd, cron) and sshd were running.=20 > Occasionally, very big jobs would come, and one or more of these processe= s=20 > would start eating up all available memory, until FreeBSD decided that it= was=20 > time to take action. For such cases, another solution exists on Linux to dynamically enlarge the swap partition (something like dphys-swapfile or SwapSpace - see ), but I do not know if such a tool exists on FreeBSD? --=20 Th. Thomas. --7RIpcGXDOQOnB0f9 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJ8BAEBCgBmBQJgNki5XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFNTM2QkU4NTM4NTM5OUQwMEI2RkFBNzZG MUM1MTZCM0M4MzU5NzUzAAoJEPHFFrPINZdTs7MP/1iAVsZAtvst8Y+1x8XqYYbD FwAbLh6xIvkRZYOfmnSARDR4/2ZN+F7ixoX30CnjtNSxaw/z1199HRlNpFlhDQDi Fc1PZyF55Z3NmHPelhlNuUPRlIcGHxP0V4bu1/RLgqX7EkdUjGbPJnTPeH7Ws2jG VAGdMBuaaL8ZehZlyTp5J7mGShWJQV3kH/7KvnNL4UZKSNUtpXe9sbeSwF78xu/r twvgRKk//7731ZMi4jwaWqHgAneCoXA4d9DHLFMnwAC4Og9r+KCHlTuE0DpE9zCB EHcXShTdcl+h3I7aALI2D4m3Xkd+t8q7nnNaCKPlE+KeFTOhstB31u+aWdDd1UBn /XioAYx4prGMxoZTuFPlYhxPUO9r6n/AUhMgLouNlq3uS9krncz5E6w3aeuLvEG4 4ipYkGlu60iRKrX9x8ZMltBVEZ6sO74AdT2Tr5v/Y88TBhiWicymSlBvSDKBeQnb DkuQSBInb6SOYkooIQbQfxyjopQQ37UvWg/UqM/FXwJgl/UtlbKzVaaDZWfIPfZB QAJqKkcMKW02PCXauFgrcszrfkiGKARoB6SkCZjJW38ldCUTO9LbZOEPekRedAi8 r6tacc+66BodRcpwQPDNSNu48ZGfomEbeOyaD5CJeb2uzUOLapAXhTLLrhL9ymhe H6jqFUWx2jt4qulg+Ezd =3+8/ -----END PGP SIGNATURE----- --7RIpcGXDOQOnB0f9-- From owner-freebsd-hackers@freebsd.org Wed Feb 24 14:26:29 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 97D9355E958 for ; Wed, 24 Feb 2021 14:26:29 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by mx1.freebsd.org (Postfix) with ESMTP id 4DlysJ5XX1z3LRd; Wed, 24 Feb 2021 14:26:28 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from [194.32.164.25] ([194.32.164.25]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id 11OEQLil086557; Wed, 24 Feb 2021 14:26:21 GMT (envelope-from rb@gid.co.uk) From: Bob Bishop Message-Id: <500F4BE2-87C4-45F8-85A9-326498F47CC5@gid.co.uk> Content-Type: multipart/signed; boundary="Apple-Mail=_CDE4075F-BFD0-4BB3-965F-0290FF30EDF0"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: The out-of-swap killer makes poor choices Date: Wed, 24 Feb 2021 14:26:20 +0000 In-Reply-To: Cc: freebsd-hackers@freebsd.org To: Thierry Thomas References: <1984125.0OzZcVfBr4@ravel> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4DlysJ5XX1z3LRd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rb@gid.co.uk designates 194.32.164.250 as permitted sender) smtp.mailfrom=rb@gid.co.uk X-Spamd-Result: default: False [-4.80 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; HAS_ATTACHMENT(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[194.32.164.250:from]; R_SPF_ALLOW(-0.20)[+mx]; SPAMHAUS_ZRD(0.00)[194.32.164.250:from:127.0.2.255]; DMARC_NA(0.00)[gid.co.uk]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-0.999]; SIGNED_PGP(-2.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:42831, ipnet:194.32.164.0/24, country:GB]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 14:26:29 -0000 --Apple-Mail=_CDE4075F-BFD0-4BB3-965F-0290FF30EDF0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, > On 24 Feb 2021, at 12:38, Thierry Thomas wrote: >=20 > Le mer. 24 f=C3=A9vr. 21 =C3=A0 13:02:14 +0100, Olivier Certner = > =C3=A9crivait : >=20 >> Hi, >=20 > Hello, >=20 >> 2 to 3 years ago, I stumbled against production problems on servers = doing >> heavy computations. Only a few processes (2 generally) were doing = them, and >> most of the time consumed less than 1/4 of the available RAM (2 GiB). = Apart >> from that, no other process was allocating any significant amount of = memory. >> Only some base default daemons (syslogd, cron) and sshd were running. >> Occasionally, very big jobs would come, and one or more of these = processes >> would start eating up all available memory, until FreeBSD decided = that it was >> time to take action. >=20 > For such cases, another solution exists on Linux to dynamically = enlarge > the swap partition (something like dphys-swapfile or SwapSpace - see > ), but I do not know if such a > tool exists on FreeBSD? You can of course add swap to a running system using swapon(8), but I = have no idea whether this will improve behaviour if the system is = already low on resources. > -- > Th. Thomas. -- Bob Bishop rb@gid.co.uk --Apple-Mail=_CDE4075F-BFD0-4BB3-965F-0290FF30EDF0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQR+a6Wh87I/iYwcbE+8xpPppLfFvwUCYDZiDAAKCRC8xpPppLfF v6H2AKCGszzAPANOMH0st0NkTjrNc1VcZQCfcK2VMcf0P9pIsLZuR5xnHx2u6qs= =sT/v -----END PGP SIGNATURE----- --Apple-Mail=_CDE4075F-BFD0-4BB3-965F-0290FF30EDF0-- From owner-freebsd-hackers@freebsd.org Wed Feb 24 15:26:32 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 26DC95603C7 for ; Wed, 24 Feb 2021 15:26:32 +0000 (UTC) (envelope-from Walter.von.Entferndt@posteo.net) Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm0Bb2Phdz3Qbs for ; Wed, 24 Feb 2021 15:26:31 +0000 (UTC) (envelope-from Walter.von.Entferndt@posteo.net) Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id A563D160060 for ; Wed, 24 Feb 2021 16:26:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1614180388; bh=pZPhE3pIEhHh+Rmi941y/hnvNs+W5CyZdXRaZdefxe4=; h=From:To:Cc:Subject:Date:From; b=J3MudocWDhyfkZ/6CJp2Ff0mjzhtll/xSdeN+Urry8zBJAlS072o2lzt4dBAaoWp6 96apbjQENYFKKCHtTWE3GP68QNjkKlkmhcOq5eD1SBD/xqiZnDQ2XidENT6hwZdKGC pio/LcaK09NLmC+uOZm4sB9LUZkkleUI22ayGoPT/ibzWElPpOAgHZjx5lOnKh8TyT weBkiCUPwSvfv7e6od76fMHVaY5J8cdgSO3z3URK9myzO7vZHPbUlUUQP0Te9vIPz6 hWyfL2o8WyOAhvy38n/AVLWlL8tMryyGDMw8Xp5QOCuR93cXpN2wrEa9K4TkAcnGqi bnLbXjC2nXOKw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Dm0BW4vK4z6tmd; Wed, 24 Feb 2021 16:26:27 +0100 (CET) From: Walter von Entferndt To: freebsd-hackers@freebsd.org Cc: Alan Somers , Konstantin Belousov , Mark Millard Subject: Re: The out-of-swap killer makes poor choices Date: Wed, 24 Feb 2021 16:26:16 +0100 Message-ID: <9084784.RH3biPoPvx@t450s.local.lan> X-Face: #$[hC+4[4W*mS3hB&izisyT_#E]^Aq+7Isv`2Tu5q*1~jR@&['74B>Ibyrk]GTJ!j$ NjX=#L2#k2X7OnaaRM_Pd5`>`8OJ3; +I2 References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 4Dm0Bb2Phdz3Qbs X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=posteo.net header.s=2017 header.b=J3MudocW; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (mx1.freebsd.org: domain of Walter.von.Entferndt@posteo.net designates 185.67.36.65 as permitted sender) smtp.mailfrom=Walter.von.Entferndt@posteo.net X-Spamd-Result: default: False [-4.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:185.67.36.0/23]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_IN_DNSWL_MED(-0.20)[185.67.36.65:from]; DKIM_TRACE(0.00)[posteo.net:+]; DMARC_POLICY_ALLOW(-0.50)[posteo.net,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; CTE_CASE(0.50)[]; ASN(0.00)[asn:8495, ipnet:185.67.36.0/23, country:DE]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[posteo.net:s=2017]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[185.67.36.65:from]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com,yahoo.com]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7Bit X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 15:26:32 -0000 I'd like to suggest that it might be useful to add more (simple) criteria to apply ordered classes/queues to the list of processes that are candidates for beeing killed, similar to the time scheduler's runtime queues. Then the (e)uid of the process comes to mind. It's not expensive to check the processes' uid (or euid) and favour the so-called system users (usually uid <= 1000) over other (human) users; i.e. apply the order: [humans] < [system] < [trad. daemons (uid <124)] < [root]. Then kill from the lowest queue (humans) before the next etc.pp. Or use these classes to apply a weight to the processes' page count. OTT, maybe it's ok to kill jails (or processes therein) before the host. Such an addition could be simple, but efficiently give better choices. Regards -- =|o) "Stell' Dir vor es geht und keiner kriegt's hin." (Wolfgang Neuss) From owner-freebsd-hackers@freebsd.org Wed Feb 24 17:00:08 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 18EFB5625EC for ; Wed, 24 Feb 2021 17:00:08 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from edna.lautre.net (edna.lautre.net [80.67.160.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "lautre.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm2Gb2XZkz3mnj for ; Wed, 24 Feb 2021 17:00:07 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from graf.pompo.net (graf.pompo.net [78.225.128.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by edna.lautre.net (Postfix) with ESMTPSA id AE27E11D90C for ; Wed, 24 Feb 2021 18:00:02 +0100 (CET) Received: by graf.pompo.net (Postfix, from userid 1001) id B9B467CAA7E; Wed, 24 Feb 2021 17:59:56 +0100 (CET) Date: Wed, 24 Feb 2021 17:59:56 +0100 From: Thierry Thomas To: freebsd-hackers@freebsd.org Subject: Re: The out-of-swap killer makes poor choices Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <1984125.0OzZcVfBr4@ravel> <500F4BE2-87C4-45F8-85A9-326498F47CC5@gid.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <500F4BE2-87C4-45F8-85A9-326498F47CC5@gid.co.uk> X-Operating-System: FreeBSD 12.2-STABLE amd64 Organization: Kabbale Eros X-Face: (hRbQnK~Pt7$ct`!fupO(`y_WL4^-Iwn4@ly-.,[4xC4xc; y=\ipKMNm<1J>lv@PP~7Z<.t KjAnXLs: X-PGP: 0xF1C516B3C8359753 X-Rspamd-Queue-Id: 4Dm2Gb2XZkz3mnj X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of thierry@pompo.net designates 80.67.160.88 as permitted sender) smtp.mailfrom=thierry@pompo.net X-Spamd-Result: default: False [-1.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; FORGED_SENDER(0.30)[thierry@freebsd.org,thierry@pompo.net]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[80.67.160.88:from]; ASN(0.00)[asn:20766, ipnet:80.67.160.0/19, country:FR]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[thierry@freebsd.org,thierry@pompo.net]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[thierry]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[80.67.160.88:from:127.0.2.255]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 17:00:08 -0000 Le mer. 24 févr. 21 à 15:26:20 +0100, Bob Bishop écrivait : > > For such cases, another solution exists on Linux to dynamically enlarge > > the swap partition (something like dphys-swapfile or SwapSpace - see > > ), but I do not know if such a > > tool exists on FreeBSD? > > You can of course add swap to a running system using swapon(8), but I > have no idea whether this will improve behaviour if the system is > already low on resources. But swapon requires a manual action, this is not dynamically adjusted. -- Th. Thomas. From owner-freebsd-hackers@freebsd.org Wed Feb 24 17:04:16 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7D64C562CB2 for ; Wed, 24 Feb 2021 17:04:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm2MM5R74z3nLt for ; Wed, 24 Feb 2021 17:04:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qv1-xf2e.google.com with SMTP id cw15so574840qvb.11 for ; Wed, 24 Feb 2021 09:04:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=54tCw4bHZdxwmHL/lo0n7DjPMSkabebCLjBcFQUH9aY=; b=a+2fVO7thzwDakWINjRJwQW68hgL0qv36u1dAESJKwESBhzksy1sHzD/2dIzpPbsXx 7zDhzvqywBO2GdpTiWnoHrS+eTQDlT0RqzfZcl8ZKurl9HdzX0ZEdD3FioDm18GIcFLq 6NJ3GLwynWtDj1hMlWAD+6cSn5ITr2qYW6PertoxdmfQuRyFJ4oC8k40Hak0G+4h5D4s 9JWdlpdaWC6oDO6Km4+jvVqdj0UgctUdNfCPXNFxdb6bhazmp/Pq/QPItZ0in5uuV43d zpEUh0UDsV+AE/ACb0qmZ+8EGOSJCXlcz4EzOnGCtoJ4r+vskwN0j5a7hRAoaFD25sya c0+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=54tCw4bHZdxwmHL/lo0n7DjPMSkabebCLjBcFQUH9aY=; b=gdXd0dg2HZdgLegp+BHEaDNj9FX3wik9rUJRXYdg1rc44VQ0A/nSVyGr+FtPy2DjF3 EuPcwuLOlXoVAiJ6m1sr0XmheLpSG7UH56AzclmO5G5YGkJnbchfLQCFWpXBw8oyd7NC IwRdsu6hvQW+3OzVKq3r2FgX96qRr6cK0YJoc37JxH3bw71XAuXO/zETl9JYgS0PvYux um7S4aFOUaxSGShxsdDCEWKnzPNSc0xoR+AFnHIP6VN2mWgsBs9QI700lDVOgokls4HZ P4rN7WgvPRWNYnxzNfidSmyiHxjCUl5zyym20SKFSJo8bZkoUS9f5NlKBaLDzltdQ02o MIjQ== X-Gm-Message-State: AOAM532WfJDhBP+1yImxRredGEY6WC4W68ehMDDDEaaag0GZ7JQ/mn5F 4xJ8yXeOusihH9mfuOSnbXZ5pStk1p/NZPe+cllfXcrRRvM= X-Google-Smtp-Source: ABdhPJyG2SnxNwMUXqvf4DUpt5/CiVqFBBYeqebWnyOj3uNrAQPOGsCxbZeRYWWJH+T65iY8F7IQNpyT9Mpm1I6emuQ= X-Received: by 2002:a05:6214:b11:: with SMTP id u17mr31139900qvj.50.1614186254387; Wed, 24 Feb 2021 09:04:14 -0800 (PST) MIME-Version: 1.0 References: <1984125.0OzZcVfBr4@ravel> <500F4BE2-87C4-45F8-85A9-326498F47CC5@gid.co.uk> In-Reply-To: From: Freddie Cash Date: Wed, 24 Feb 2021 09:04:03 -0800 Message-ID: Subject: Re: The out-of-swap killer makes poor choices To: FreeBSD Hackers X-Rspamd-Queue-Id: 4Dm2MM5R74z3nLt X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=a+2fVO7t; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of fjwcash@gmail.com designates 2607:f8b0:4864:20::f2e as permitted sender) smtp.mailfrom=fjwcash@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::f2e:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::f2e:from:127.0.2.255]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f2e:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 17:04:16 -0000 On Wed, Feb 24, 2021 at 9:00 AM Thierry Thomas wrote: > Le mer. 24 f=C3=A9vr. 21 =C3=A0 15:26:20 +0100, Bob Bishop > =C3=A9crivait : > > > > For such cases, another solution exists on Linux to dynamically enlar= ge > > > the swap partition (something like dphys-swapfile or SwapSpace - see > > > ), but I do not know if such = a > > > tool exists on FreeBSD? > > > > You can of course add swap to a running system using swapon(8), but I > > have no idea whether this will improve behaviour if the system is > > already low on resources. > > But swapon requires a manual action, this is not dynamically adjusted. > Isn't that what the sysutils/swapexd port does? https://www.freshports.org/sysutils/swapexd/ --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-hackers@freebsd.org Wed Feb 24 17:22:56 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D9D245631BF for ; Wed, 24 Feb 2021 17:22:56 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from edna.lautre.net (edna.lautre.net [80.67.160.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "lautre.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm2mw1XkGz3ppt for ; Wed, 24 Feb 2021 17:22:56 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from graf.pompo.net (graf.pompo.net [78.225.128.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by edna.lautre.net (Postfix) with ESMTPSA id C9C0E11D968 for ; Wed, 24 Feb 2021 18:22:54 +0100 (CET) Received: by graf.pompo.net (Postfix, from userid 1001) id 19C577CAE23; Wed, 24 Feb 2021 18:22:54 +0100 (CET) Date: Wed, 24 Feb 2021 18:22:54 +0100 From: Thierry Thomas To: freebsd-hackers@freebsd.org Subject: Re: The out-of-swap killer makes poor choices Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <1984125.0OzZcVfBr4@ravel> <500F4BE2-87C4-45F8-85A9-326498F47CC5@gid.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Operating-System: FreeBSD 12.2-STABLE amd64 Organization: Kabbale Eros X-Face: (hRbQnK~Pt7$ct`!fupO(`y_WL4^-Iwn4@ly-.,[4xC4xc; y=\ipKMNm<1J>lv@PP~7Z<.t KjAnXLs: X-PGP: 0xF1C516B3C8359753 X-Rspamd-Queue-Id: 4Dm2mw1XkGz3ppt X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of thierry@pompo.net designates 80.67.160.88 as permitted sender) smtp.mailfrom=thierry@pompo.net X-Spamd-Result: default: False [-2.88 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; NEURAL_HAM_SHORT(-0.88)[-0.881]; FORGED_SENDER(0.30)[thierry@freebsd.org,thierry@pompo.net]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[80.67.160.88:from]; ASN(0.00)[asn:20766, ipnet:80.67.160.0/19, country:FR]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[thierry@freebsd.org,thierry@pompo.net]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[thierry]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[80.67.160.88:from:127.0.2.255]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 17:22:56 -0000 Le mer. 24 févr. 21 à 18:04:03 +0100, Freddie Cash écrivait : > > But swapon requires a manual action, this is not dynamically adjusted. > > > > Isn't that what the sysutils/swapexd port does? > > https://www.freshports.org/sysutils/swapexd/ Great! Now I'll have to test it ;-) -- Th. Thomas. From owner-freebsd-hackers@freebsd.org Wed Feb 24 17:34:35 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E2C1C563826 for ; Wed, 24 Feb 2021 17:34:35 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm32M60rRz3qs1 for ; Wed, 24 Feb 2021 17:34:35 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oo1-f50.google.com with SMTP id s10so704635oom.6 for ; Wed, 24 Feb 2021 09:34:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zk8g2TH2JwI+kgI/xqAAq2GqOY/sWYNuxO493ye6AVo=; b=YbhbZk7OW02YAJhcMkkKTorG/PmgcJLrfWPJR/zsgPuiVgcAv47ttqxmpBORgVFgMS vV9ucJWdS7K7ZNQ0ze1VPAY3swqapvCg/5vYc0abppYRsK8l0lVLUKhnPTsJXr0haWhp QozyEaVjdX7NE9LolaOrwrqDs872Xf5cON9yDudhH0NSx0F/BY/mDKJZjPV+BCCUV8By 4pO33XL61R14caDHHf2rgsVT7x3M8B8XNu6d+dDuiGzFHJ6T7wYZqaLtsSfNJIkBlTVr +weGeSSeYerYPOsFRVMxoaY/xTVVEr+csxLnfIOvF2fR8cgpP/K/WN4x+1DkOVk6jiR+ xKyA== X-Gm-Message-State: AOAM530rPB7GrLmssYZV55BOdCWbV6UW7V6f3NHh0Zr3wAxuy5ORgpLY csK9lt5yrTWynM2cPDDhi26jU4MxrM2u1H2dLMw= X-Google-Smtp-Source: ABdhPJwZ23Fy6smd3YzSUW5eVl31JAuhmDAgL6gr/jeJbbu3uJzuG+nZuewIYRTSzS/VhKVJiEcEJ3Pijn8yZBiRUWw= X-Received: by 2002:a4a:970b:: with SMTP id u11mr24835975ooi.79.1614188074723; Wed, 24 Feb 2021 09:34:34 -0800 (PST) MIME-Version: 1.0 References: <1984125.0OzZcVfBr4@ravel> In-Reply-To: <1984125.0OzZcVfBr4@ravel> From: Alan Somers Date: Wed, 24 Feb 2021 10:34:23 -0700 Message-ID: Subject: Re: The out-of-swap killer makes poor choices To: Olivier Certner Cc: Konstantin Belousov , FreeBSD Hackers X-Rspamd-Queue-Id: 4Dm32M60rRz3qs1 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 17:34:35 -0000 On Wed, Feb 24, 2021 at 5:02 AM Olivier Certner wrote: > Hi, > > > > Ok, I'll abandon this idea. > > I hope you don't abandon the idea of improving the OOM killer in the long > term > if you feel that something is wrong. > > > I explained the reasoning for the current design, even if it actually > > evolved this way, instead being written as a whole with the stated goal. > > I do not object against adding something that would help to get it more > > fit with different goals as well, but the current idea of making the > > system survive should be kept. > > So true. The main goal (system survival) does not prevent (not so) > secondary > ones. > > I'm sorry not to have any technical contribution to propose, but instead I > have some testimony that may interest you, although old. > > 2 to 3 years ago, I stumbled against production problems on servers doing > heavy computations. Only a few processes (2 generally) were doing them, > and > most of the time consumed less than 1/4 of the available RAM (2 GiB). > Apart > from that, no other process was allocating any significant amount of > memory. > Only some base default daemons (syslogd, cron) and sshd were running. > Occasionally, very big jobs would come, and one or more of these processes > would start eating up all available memory, until FreeBSD decided that it > was > time to take action. > > Sometimes it would decide to kill one of these processes, but more often > than > not sshd or cron were killed instead, although they were consuming > ridiculous > amounts of memory. I tried tweaks via vm.pageout_oom_seq (I think I set it > to > 120, as Mark did) and vm.pfault_oom_attempts, without much change. > > In the end, I decided to use 'protect', via rc.conf's '*_oomprotect="YES"' > facility, to workaround this problem and save me some headaches. > > At some point, some of these machines had swap configured (separate AWS > disks), but later I removed swap entirely. What I report occurred for the > latter configuration, but IIRC I observed similar behavior in the former. > > I have not had this use case since then, so I can't say if this has been > fixed > (by commits such as r353734/d307bdcc2c473858) or not. > > -- > Olivier Certner > There's another silly problem that I didn't mention in my original post. The old rule of thumb is that the swap partition's size should be twice as large as the amount of RAM. However, that's no longer possible in many cases. The kernel imposes a hard limit of 64 GiB (on amd64 at least) on the usable size of any swap partition, and many servers now have far more than 64 GiB of RAM. So the advice needs to change with the times. I don't know what the best size would be for a modern server, but I would guess that it must be at least several times the RSS of your largest process, and also at least one tenth of RAM (for use as a dump device with compressed core dumps). -Alan From owner-freebsd-hackers@freebsd.org Wed Feb 24 18:08:44 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BDAD7564924 for ; Wed, 24 Feb 2021 18:08:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm3nl5KrBz3tPp for ; Wed, 24 Feb 2021 18:08:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1614190120; bh=WbFJxD/dXeaw2kiQmyrvT+NhyYtzX7//iXu3rfIPPaO=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=gsBbCiTJwKz7FEteaL6fmrzSSs14mawIbP77DtgKc22Wyrdwb4ltqvimaIDHlTjxpwStzdrPgvOczMIVuiY3UoO/ZzfXHecry9w++e/CvzRZsI7vDFzOMUvoUeOHAQTimIgWcgJZI1t2iv+9qkoPkspi//7t5VwGmC8WkuJhE4p2HQrybMoNDOz5D7NV1ksirsQS2QB5x2flIXgdh/BpBWC99kInpU3HlkBHojUFTTS4rhQlvc26Lp8aA9SjBMxpqJngFob0fLXyHRN+MCFQCCcJ01soKgzw+L26n4Ag5WDtCZB6yPGwkpUAG3z/Hxk0xecbmOYLG4AdFVVsvhiq1Q== X-YMail-OSG: pTA0994VM1krWGP092I0X4nhxDwGk3s4rzMGVBvvp7p.SnCaIE5ldyMNEqfIqv0 VGfaq62oh44TAn9Uul6CSWwCs0FKlGTsCaE_g4U9zwqK_TAXKiyASx3jyPj8FvzlkamBzTc8JyS6 k8FKzUjka4lCXpjcWaNX0uW8pfkV4P7t9neIq8y5yTi0B.CK6ZlKcswDzS.A.JvgOcF_IwOOEXFA 76L1FBhxPN8zjtCvUrkZhq1pS1c0e_N91RTG7dgrhQEA_InIvpii2PbV8PuikTHF.ENhXrL8gzuJ qNQaWFjfF7VReETW5jokwFv_0Bl03EzKA2muSgDQisHqYaLcCicsYD5ePc1X_Gey8j3B8d35GP9N SiqP2tufVy16E6FlnRjHFLsM67GQU2xAFRIrtzalMxxUjC4E2O3njY2922Uko5Tjt.JOgkgD_4IL DSBVCsfW8Tc0iIwjQQ_ee_rBD3QKGhpksyjpfs4RH92aeK3BMB5RmNPuHR0cqkp.q73yIQvDUJtI caT7BP9raIDg9QCbLxCbavWGoFEfZ7.52KDjphDRh3scYHUZwA1Plo4EN7Ey0dXrDUOKbOpZnXuJ iv4vETN_9m0Ebbx4wtqM2UhyUAv_EA4inB7gxQLDSKfa_jtpfYjt6jqfSFSlWggb4V5grR4EGhtv _ByToJ7gs41aLTUyAH4dsBB0fg150sLrHqvz8gIMS94NNmLlouFXT.xRu6MKvXoQKFn0MwO0y9cZ 5ZWa0BxhqRYDF1pVZeyuCquYf1b4GjkETBHoxAf71gM0k17os6kiK.0gUs9URvoI1fgcE5x2v8a9 7j9ewnyePBxvV0ICSLHO99JILBCeEQHKd2pXREO8LieuCR0OWcWjjyViucB9FO_BmzjkhUI1wzaw pdQl8EWofmZRBsHy.i0hySFE0fCwtDwkebb4zNUPx77kSIcB7YtnLqom37ax6K7ChZIMedO2IGnD eiqe9UE7Qb.SO48bVT8MALXFzWdC8s4BYo8YbfTY4XtEsFhqwpEFZaaf2CDXIv_wgBQQvkpN3zv8 XKj0mi5uE64CoCp6GnPpBrovOwpJM0z5.nrEuankW4A92wiWDYKENOnW2lDXJKftXWqeYYXEv3Fr 86r5CwcxzEX9N4ZxofgUetE15CKrGHpHlDEoiEQqpY4lG36dm6B3ffSDTXidHMW4.lRvkB4Swbfp 0WTXt9AMQQycLlULFGPYsRkUzc7WTnC7CVNdtdny.Xrx3BrxkULxaBRlGojATbEkQ9SbiR1F_nyL 0XAlCny3PLv5.790RxDRq9f6I1BDsJXRtAYwA983DXSNt9316bcIP3x_m8HNbOMf4ebn98ei6q5c b5fplNYHYdeF8G0r3.M7UB_JMwhVg.aSf31S0oT8BeYTTrnkCZjp_zJHog.yiLDjnF.Skp1_W47K prQAqfUoWUAl4YMkNXXvrLpSzHi2Dy_2qn5896Fi8nslETgeMMIsFeX7DYyatYY8BBoDK4qz0JvH SAcbKF467MaWGq9u6Q7FP_OSeqN.2jtQRB_nAkcAhjpJ9pXHgVjaikFEJcXLWUgvI9OIg50OO3u7 SNm07Raw.KqKXj0FaiWit3c6E4QHyvtZ.lRf9j0hdI58XNgG2Wse1wvJGff6wDiCuzOEoROyEaVx Kb76cqiX.KRGWgw.lY3fx9efs_pPnlXSJ2dWCNLGju6sDy0go9vJVrAzlphF0vSkCYmcyszsTpk_ FGsCwxugwlVROR3KPx24GhTBrBJdwj0.sOe8O9VIP5CtbP.XcFVBTpZw1xdc2KMNHCgqpC.vfVLg 4da6xEijdcPk2e63fP_8TXVXDmXLapCD.50pGeT5r0aFGfGUmIp4qwR0aKHO9vsBb8jH5zIDuMVB GmAbEBmQXb3y2t5OnkU72jWxbrrPFjk6IfkLmKEAB6ZVTzKiZ8wkJIm0rPAQiN88_05vLiNFgE05 7fq8p2O3Y0aFAEqp.q6H3EhKMdaISv_wdVMSvYiesB9SjW9u_nHDbUspA9e.0n3BvJ.nfPEkkOzV X9nyYzcb5exfUw0xP478xA1O6qYI3JqxKZozqKPjw3CWBHddK2Us8pMl5vXwtcIiCu57eMJhIvRj 1mnyDQB5rDuovkK4MN46rNeP_cxQJFogtuLrnfS5BJTmxJV2t0YvIp6vBmS8hvce7KGYV.21HNyh _M1K.WC2i5MNgx.rGMLtzwDDygTHQvrQJ7eJVpw5rM9whAttuKsRjz5igYvGIxwVq6Mh.L9zhidu y4R7gCbIioBF.iGiRBMbwe9Mu3nFq4BiJ6XBVyjKWdgimnRykmK12vakq1SSneaBcXYKlwfjTJRF ixSwiyvQ1lpA6zkx32ViIDjG1AiLRHdKpKXVeGgwsoXkK.p7GkUxHxxwN2eVNOsmBbLN2g46q1kr d6LJ_lXBZn0s_Qgtbk1uT5QijEbvcwVP2_o3o8kyxcV_dab_ef27d_7OFel4hhmJv4sB31UVXCvH D_KgmiGAiObGHUR5WSeJuDsEHR6k6BgTFGJ7K2AE2FHfKivfW.13mwQz6000x7SPFULt3szZmhpp NPE2m.mxrFLm.oaMEDqdhLJT5CjREFghE61RpFI8qmsF.3XxP0ZXPhIDi7vy9NQqHNv.JuDAFNsm yoYKUM44LZsDGUR5uRmnnXygSUwQWXTWRtpew.y9xGDknSfKr5ZPW9GqDpgxsssvJ6isR7jyOLRD dDpEZByHyYOYv6dNIKWZBQuazy9Y- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Wed, 24 Feb 2021 18:08:40 +0000 Received: by kubenode552.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 9a58245c4e1ca08325660ad8a9aba0a4; Wed, 24 Feb 2021 18:08:38 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: The out-of-swap killer makes poor choices From: Mark Millard In-Reply-To: Date: Wed, 24 Feb 2021 10:08:37 -0800 Cc: Olivier Certner , Konstantin Belousov , FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <96B17A3F-4663-42E2-9D29-C022803C9864@yahoo.com> References: <1984125.0OzZcVfBr4@ravel> To: Alan Somers X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4Dm3nl5KrBz3tPp X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.64.83:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.64.83:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.83:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.83:from]; FREEMAIL_CC(0.00)[free.fr,gmail.com,freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 18:08:44 -0000 On 2021-Feb-24, at 09:34, Alan Somers wrote: >> . . . >=20 > There's another silly problem that I didn't mention in my original = post. > The old rule of thumb is that the swap partition's size should be = twice as > large as the amount of RAM. However, that's no longer possible in = many > cases. The kernel imposes a hard limit of 64 GiB (on amd64 at least) = on > the usable size of any swap partition, and many servers now have far = more > than 64 GiB of RAM. So the advice needs to change with the times. I = don't > know what the best size would be for a modern server, but I would = guess > that it must be at least several times the RSS of your largest = process, and > also at least one tenth of RAM (for use as a dump device with = compressed > core dumps). That was fixed in main at least: # swapinfo Device 1K-blocks Used Avail Capacity /dev/gpt/FBSDFHUGEswap 201326592 0 201326592 0% That is from a system based on main 3acea07c1873 : # ~/fbsd-based-on-what-freebsd-main.sh=20 merge-base: 3acea07c1873b1e4042f4a4fa8668745ee59f15b merge-base: CommitDate: 2021-02-08 19:15:21 +0000 c1845d00f818 (HEAD -> mm-src) mm-src snapshot for mm's patched build in = git context. 3acea07c1873 (pure-src) Restore the augmented strlen commentary FreeBSD FBSDFHUGE 14.0-CURRENT FreeBSD 14.0-CURRENT = mm-src-n244686-c1845d00f818 GENERIC-NODBG amd64 amd64 1400004 1400004 But the change is older than that. There was a period where I had three 64 GiByte swap partitions instead of one 192 GiByte swap partition, because each was forcibly limited in size but the total was not (on the scale involved in the example context). =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Wed Feb 24 18:11:14 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5237F564ADA for ; Wed, 24 Feb 2021 18:11:14 +0000 (UTC) (envelope-from olivier.freebsd@free.fr) Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [IPv6:2a01:e0c:1:1599::15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm3rZ5yvTz3v4Y; Wed, 24 Feb 2021 18:11:10 +0000 (UTC) (envelope-from olivier.freebsd@free.fr) Received: from ravel.localnet (unknown [90.118.181.206]) (Authenticated sender: olivier.freebsd@free.fr) by smtp6-g21.free.fr (Postfix) with ESMTPSA id 93DCD7802C5; Wed, 24 Feb 2021 19:11:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=free.fr; s=smtp-20201208; t=1614190267; bh=wOcEHQcZIFt/pP6zPFjRl6W5AE46niyVfDplHYf+mJw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Xp3Kwaj2jNmhdka8OQJ3YNbs8lfyRIarMr6PZH7yIfAbpc52GTvSFPtM/FZzk6ROU fb4LWzdLpEUyr7/ogoKj4MDFnCoGlpRn1djIRErgSt0Jsg4pzcXWQq5qKd8IooV0YA GBxBTuvWC2635FwrUfvHgkFc7uVvZe8tHTJ2K7mU+XczyG1EDdvM+wMcvahsXPwa6T ZB/5GHTB6oSpUeehfo9a7T/oLiWnjskjsi8X1+wvgtJROL2mWKqJHHjpyRDcDfXmdU rqPkKhHD7p5mxOOBE08dK7+YY2sI/9JS8DI12eZ22FSDhauFmlXYE+8LKUzSEo+xNZ kCIP991bhwPEQ== From: Olivier Certner To: Thierry Thomas Cc: freebsd-hackers@freebsd.org Subject: Re: The out-of-swap killer makes poor choices Date: Wed, 24 Feb 2021 19:11:04 +0100 Message-ID: <6928973.CEzQG83mar@ravel> In-Reply-To: References: <1984125.0OzZcVfBr4@ravel> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Dm3rZ5yvTz3v4Y X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=free.fr header.s=smtp-20201208 header.b=Xp3Kwaj2; dmarc=pass (policy=none) header.from=free.fr; spf=pass (mx1.freebsd.org: domain of olivier.freebsd@free.fr designates 2a01:e0c:1:1599::15 as permitted sender) smtp.mailfrom=olivier.freebsd@free.fr X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:e0c:1:1599::15:c]; FREEMAIL_FROM(0.00)[free.fr]; DKIM_TRACE(0.00)[free.fr:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[free.fr,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; CTE_CASE(0.50)[]; FREEMAIL_ENVFROM(0.00)[free.fr]; ASN(0.00)[asn:12322, ipnet:2a01:e00::/26, country:FR]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:e0c:1:1599::15:from]; DWL_DNSWL_NONE(0.00)[free.fr:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[free.fr:s=smtp-20201208]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[2a01:e0c:1:1599::15:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2a01:e0c:1:1599::15:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 18:11:14 -0000 > For such cases, another solution exists on Linux to dynamically enlarge > the swap partition (something like dphys-swapfile or SwapSpace - see > ), but I do not know if such a > tool exists on FreeBSD? In case there is some misunderstanding: I didn't want that in my particular case. Putting the machines to a crawl for jobs that mostly were too big anyway to survive even with swap indeed proved out to be a very bad idea. Also, removing swap was for me an occasion to test whether the absence of it was causing any change in the odd OOM killer behavior I was observing (unfortunately, as said, this was not the case). That said, I wasn't aware of these tools. Interesting to know. Thanks. -- Olivier Certner From owner-freebsd-hackers@freebsd.org Wed Feb 24 18:36:17 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC4E15657E8 for ; Wed, 24 Feb 2021 18:36:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm4PX3VLRz4Rxw; Wed, 24 Feb 2021 18:36:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11OIa70A036073 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 24 Feb 2021 20:36:10 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11OIa70A036073 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11OIa7Js036072; Wed, 24 Feb 2021 20:36:07 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 24 Feb 2021 20:36:07 +0200 From: Konstantin Belousov To: Alan Somers Cc: Olivier Certner , FreeBSD Hackers Subject: Re: The out-of-swap killer makes poor choices Message-ID: References: <1984125.0OzZcVfBr4@ravel> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4Dm4PX3VLRz4Rxw X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_CC(0.00)[free.fr,freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; R_SPF_SOFTFAIL(0.00)[~all]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-hackers]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 18:36:17 -0000 On Wed, Feb 24, 2021 at 10:34:23AM -0700, Alan Somers wrote: > There's another silly problem that I didn't mention in my original post. > The old rule of thumb is that the swap partition's size should be twice as > large as the amount of RAM. However, that's no longer possible in many > cases. The kernel imposes a hard limit of 64 GiB (on amd64 at least) on > the usable size of any swap partition, and many servers now have far more > than 64 GiB of RAM. So the advice needs to change with the times. I don't I do not think so. The usable size of the swap is determined by the amount of swap metadata we pre-configure at boot time. Usually it is sized proportionally to the available physical memory, but you can override swap zones size manually with the knob. > know what the best size would be for a modern server, but I would guess > that it must be at least several times the RSS of your largest process, and > also at least one tenth of RAM (for use as a dump device with compressed > core dumps). > -Alan From owner-freebsd-hackers@freebsd.org Wed Feb 24 19:59:42 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AD1F9567B63 for ; Wed, 24 Feb 2021 19:59:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dm6Fp306tz4ZJX for ; Wed, 24 Feb 2021 19:59:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1614196780; bh=ikOyfTTcOHd4QfvtGJFoPnvAmZ2mIuxqFdVtZF69GER=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=kK/HN+C/6FzDI1ikI1ZIi22eGiTqmA97z/iRqoiXrjdSyapVU9TJXjIM5/st+0XSMZGyTBDmrrFJd/WrXVwXeeYA0JhVaRjtVlP2zE7DJAY/EraxC5ryGVVeR5z1XY+cIlzz+TOzbzW6WZkWXJQLC6o4vr5Q8iDDMJb6sReHaVlRoSQHkZq6jy3st/jhc0ojRIe5P85a5dbEjXk7nv3S9Mui7y1C6/6pv5XxsEkUop4QkZzgGexwMc2J5bfpxQfD87CRIL2o1wHI0nby8mco1CcyGAPx3nGi0lzTOgJlPBzG6Hzs8xUVOlq51CJYwNPSTo9Ivocq85sQy+ZD3k7vlA== X-YMail-OSG: IUA1kmAVM1lFnIjbojkxC0xUJAWtfmAORyvSE8rn4ko8STQGRFguQag2jJfOwpi xVgXN0ZwC8qC82A2WZ2U7tI22fuGAnL9ciGfPRZCGr4hx48d2f0lV5PXLXDkX85hQ1GGkUmpRa5e g1exoCmn2GqOCzxCL4gOFjB9r2UiKXY6zdE3ddctY.Omk8AZHa3zqa6U8h_Ie5Ty9UqTAYvRBV0p _Ve1z7O_GZK0LPTjnphPAiVWv0hU7VZQoBzfT_iIqmuT6QwIwCRgmo0OfxdfAA_s8d7ovWWQBzGp IbQosow6Ta9TQJmDXi4S3Ly1a_myDIRpwBUER4Drlat6M7oEE56dZmQ4Mn7ArTdGqNDVU4S.ufbR qzTzvv1fqV.kOfZAVDL1IXEMSaS.p3YIpNAmCZTGrPpARnJjacuqZESAoUhuDOeJFexbyKiVlOmd ib8zZYwgGreVVBTvZ4I4qv_Y44LVPlsQ2f7PteKTJoSxOqyzzxFGVvM5BDLUfJX3NXM7kkyWTe7K tvyfBqR.75VO2rHiTfH7SIpTH.GCNQxxWWgMGbeUKadtj1tt.VZR.mw2KP1j_AYgP04O46f1kj.m v5mqxb9w_AUNeswdWG7TUt87nvRUF263Yh0VZazDDV9NE.5osoeXJqkHMQwpxKbtR9f6W.F9jcyk 2XkIRtEGWHwBUTQRMumDMpOfkx3hrAxafHrMM4EaYNs0e0R3tVHGIrVtxielkQp6Yjx85fMO5NUD NeGqtJAQX3Zzj9akgsQvYv1nntsBm_FYTSAHc_LjzWStSZiln7Yb7zyhhaz.YO8eqn3GGLZNfsNR EFBBUeI3lOwn8EC_moP4rYHHlKs74w0HVSLdxrtkAD9Bsaky3jPKM.kI_ulZBq.RLRvP606.mwFU kk_Ri6LqhNfOisccK_JNEJyet4WfpwFR2yyu7xJR9f4X6m8bLLrtyS4zQlCVCqt7oRphSsIbQhF1 Sr3lCABe3LSHTosWzizAIwqn5PoXQ84DSecKe5HnCf4bMDuWeJ7cGjfuomS7PK2rdrdUgfi8juc4 Y92vRUyCRyrXqn_NLu9MSVon_NCMk.m3jLNO1aaW0FjZtsAHagEgngLGPbgz4tO_vZGXhFNwyHHF lH2E5gtY7kpSyz2V964x2vcKCH3IuaET.W.3rgXjpZhVUkL8rGg45ldVAOsvKRFqMOS5bLL.bfmu 3eSu8J2xOloRpIsgifk7gRQlaRoleLtawi6hVTQ9IL0gCLT0RM1UpvbyzxDwi_WRDDIpq81U8lTT VF7cditHZtlnGBXbOKi1zuhtBnaFDci73zlUQ_ZebaqYxaESnh1ZOUcbxUzp3VHscumCDS6w7lLm pa9mhOkjYe4muIG98jNVIS1cZKIcABDsrmp17u_iKVtrTDeQm3sU82B5ClmtEWfYcnuxU55bZWZW JFQsI03P4X2F_wGYi9ZDkciPgGVcj.kHA_iUZXXa4L2sBS82_O1CCMtr_2SBaqoGFQg6JMULuXdq X99Moicj.BBt4a_ZIj4gfscerVX7MV4YxOgA9rjQhEcKoVy8OJP0j.R22cwRiVmXt1jpu7EaNI8. nY32nEzZTyTIrKtE3JX4SER5ipRLPWoimX2VE9p1SpRl2YwdGNXO4LNT6g34Zls_MhEC5Ms6jxNc ZcQLcq1Ts.HZhPNmLNc0LSCuNV0hFXmC8CjuWKntubnGJcaVOWixZKwQv_CcU2LwKYdPBpVyiS2g 1vRa3MSxI8zNaUGfezthTj.MeuP4zf3O_0A_yUlgJhfJSuqVYV8Yua7gQElBsydHlDFB.zCo8PS8 kK21r0Zt3XSzS8Q49skp_b3amuApJWwvJNuNK40JYNGNdprDlrH5ZShDN7f7VuISTsJS3E0CNpQO x4bmpOJCmPNysPEosEWLHjMNPf4yrZnW_eRDxrX2WktzADtFXSE20WRSWZtk4SGmxiENIukDTmTu b60CeZ.tkh.qqXhzgAnMHtyFb84sTWjYpNi..pZjZOUIBbJhUzvfwwwI_Ql7EOSKVNIvHUSjeVjS C5Jl4lyPjpmQIFvv98ACB4zyZgVjo4dbDZ8bpxmejNuoDsWgxjWxbZGdPF_77FASIViHu0nPtMGM fzaKKhbJB41p9jdOeddX3BJEIPTgQTSxaq91t0FVkNHDaq6P_cs9nuuiUhv6KNsnMEY9WTC.Dpzs CSbb42C8p_LQ9yyM6FfqUt8wWTVdAbdNpszQ1_CrFY0tbKzSXSSbwR5WiGHiQjJbZfOhFiP6JZ8K .AFJK8OWGBzSvLXFMunZ72y3pawTFF7F_oykJPCl0QEUS5Rn.Vbv8XvUn4BdPMk_rQI0S8LTRI9c S26VX.BzcL9xLuc2olZFGRRnTL5Aef55bip0DyaQgugCxtMvP3pwFeu.fEHZMxHtebnhIj10dj6y Q0vbBqpMjkFd7tbRyHyaeY5pebpgcwYwNgJkTQ3QHgpXP9S8E10aLcGOyWTAc2B40bm25222F7of 57Qlx6yv328AJI52h2DDto9zvJXxin1DuPJ2GFyF1s_ZMFymmypo5PcHNo.czxZeea2RzQEgieH. 7RrRWE2boT1jXRL_z7vLJQr9gD7.Al172xOlIjpPJ5DaefphqcVPMxhL4RLY- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Wed, 24 Feb 2021 19:59:40 +0000 Received: by smtp410.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 695e29416172c46981f0cbdd9102670b; Wed, 24 Feb 2021 19:59:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: The out-of-swap killer makes poor choices From: Mark Millard In-Reply-To: Date: Wed, 24 Feb 2021 11:59:33 -0800 Cc: Alan Somers , FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <1984125.0OzZcVfBr4@ravel> To: Konstantin Belousov X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4Dm6Fp306tz4ZJX X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 19:59:42 -0000 On 2021-Feb-24, at 10:36, Konstantin Belousov = wrote: > On Wed, Feb 24, 2021 at 10:34:23AM -0700, Alan Somers wrote: >> There's another silly problem that I didn't mention in my original = post. >> The old rule of thumb is that the swap partition's size should be = twice as >> large as the amount of RAM. However, that's no longer possible in = many >> cases. The kernel imposes a hard limit of 64 GiB (on amd64 at least) = on >> the usable size of any swap partition, and many servers now have far = more >> than 64 GiB of RAM. So the advice needs to change with the times. I = don't > I do not think so. The usable size of the swap is determined by the > amount of swap metadata we pre-configure at boot time. Usually it is > sized proportionally to the available physical memory, but you can > override swap zones size manually with the knob. There was a period of time when the 128 GiByte RAM ThreadRipper had its previous 192 GiByte swap partition use rejected and I had to split it into 3 64 GiByte ones. Later I saw a checkin that was a correction to some calculation (vague memory) and I retried having one 192 GiByte swap partition and it was again allowed. The ability to dump to a swap partition when there was a 64 GiByte limitation with 128 GiByte of RAM had implications for the configuration. I actually arranged having a partition that was only used for dump's potential use. That took some rearrangement to form a large enough space, making other tradeoffs to do so. (I'm not sure if I can find the commit that lead to me switching back to more than 64 GiByte for a swap file on the large memory machine. I do not remember details any more.) >> know what the best size would be for a modern server, but I would = guess >> that it must be at least several times the RSS of your largest = process, and >> also at least one tenth of RAM (for use as a dump device with = compressed >> core dumps). =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Wed Feb 24 23:37:16 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7437954D009 for ; Wed, 24 Feb 2021 23:37:16 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmC4r2z7Zz4n2p for ; Wed, 24 Feb 2021 23:37:16 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from [192.168.1.10] (unknown [98.42.164.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: rpokala) by smtp.freebsd.org (Postfix) with ESMTPSA id 0B7BCF129 for ; Wed, 24 Feb 2021 23:37:15 +0000 (UTC) (envelope-from rpokala@freebsd.org) User-Agent: Microsoft-MacOutlook/16.45.21011103 Date: Wed, 24 Feb 2021 15:37:12 -0800 Subject: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko From: Ravi Pokala To: "freebsd-hackers@freebsd.org" Message-ID: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> Thread-Topic: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 23:37:16 -0000 Hi folks, A colleague and I both independently observed `sysctl -a' appear to hang on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C didn't kill it. We could still establish a new terminal session to the node, via SSH or serial console, and we were able to see that it was actually spinning, not hung, and was consuming an entire CPU. We eventually determined that it was specifically `sysctl vm.pmap.kernel_maps' which was spinning, and subsequently that it only spinned if nvdimm.ko was loaded. It was not necessary to access the device node associated with the NVDIMM; merely having the module loaded was sufficient. I know nvdimm(4) isn't terribly widely used, but hopefully someone who uses it can at least confirm my findings on this. Help in debugging would be even more appreciated. Thanks, Ravi (rpokala@) From owner-freebsd-hackers@freebsd.org Wed Feb 24 23:49:42 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 10EAB54D223 for ; Wed, 24 Feb 2021 23:49:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmCM95XtQz4nZQ; Wed, 24 Feb 2021 23:49:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11ONnQWg011804 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 25 Feb 2021 01:49:29 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11ONnQWg011804 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11ONnQqM011803; Thu, 25 Feb 2021 01:49:26 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 25 Feb 2021 01:49:26 +0200 From: Konstantin Belousov To: Ravi Pokala Cc: "freebsd-hackers@freebsd.org" Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Message-ID: References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DmCM95XtQz4nZQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 23:49:42 -0000 On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > Hi folks, > > A colleague and I both independently observed `sysctl -a' appear to hang on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C didn't kill it. We could still establish a new terminal session to the node, via SSH or serial console, and we were able to see that it was actually spinning, not hung, and was consuming an entire CPU. > > We eventually determined that it was specifically `sysctl vm.pmap.kernel_maps' which was spinning, and subsequently that it only spinned if nvdimm.ko was loaded. It was not necessary to access the device node associated with the NVDIMM; merely having the module loaded was sufficient. > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who uses it can at least confirm my findings on this. Help in debugging would be even more appreciated. > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this could be quite large. It could be busy dumping page tables. Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). From owner-freebsd-hackers@freebsd.org Wed Feb 24 23:53:42 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6474A54D51E for ; Wed, 24 Feb 2021 23:53:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmCRn2Dcqz4pHp for ; Wed, 24 Feb 2021 23:53:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1614210819; bh=5dGSKdb4FL4b1YHfkB5tlpOHYWz8BOHB7TJVFELs6Zq=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=Eu+L71PGCqmmQdTdlC/buirwCbg52OHgeLMGzwQ5LD3EnROum9JxD94nennFIw2OlFQksx7K4b86MMss5oW21nAi4NfjVASqzx04M+l3AJ04kqIQpCn5exzF25osa2QoixWNSuLz+8GyFaAVB7IvFpPVKu/ODj17NPiTNwJGtUKEStJc7tTuRk6LIi4OfR8Q6x5R263SFOO7D0aAo5fTrFA6WOC8bKx0hNnbKaTfP6GkqfywzLRXIjqkH473tGL261sZVwb1+22Do8qzqcxfff7nlj1yf2MhGrGIU5dFlyQRwx6QgQBXvGCPyyQadCrLmKeKsIYLnJvxTS/qUEU3Fw== X-YMail-OSG: KEQPjs8VM1l1rwKPBJZBAR3u0yKh_XGm2AuFtQgp3TsQFcGHDr5ypJYTLh1i7FG KKcPK3TKqbqPUUdqBRrjP1KKW0jmu_fgk0m5jAugyO2DCRQ3kqzycFXC4Ofg4qFjoyh4Q.Kf0IQ0 sWEQ48Hg6.D_D2EMC6xhIhinADVHQbHmqz9SSQfW_0f0zzsIzZFPbQ.MAHciSK_U.9W9x7WdsP5h iTVT3r2zcmmH_k8HK1MaqGhr9SVUnBgcObCDH7UNRC3QmAEdCJkFq3kcEw4.7jvv5pk2BxAf8vNK W25oF9U2KnYL4Iie4TbahJDsq_joBtayB0kUFqcTqQTchYkFCnXleQ8k07X3I.LPI7.4dlwWU3kU sseZBQu1LyvZ8K35HOSBChPkql3nSIm42at3MiMikjpY2TYBz6WMBwqG9OWWkBDuLZn1kYYCJ5p_ sgemrJ71fcKLmNNbYtcoBW6L.fHpGEPUeTw7PJvgL_rT4t3RlTNqu0r2UwYKaWLmY7ik05KKgCPe ojNVsw4tTWUZO6ZwyZPz_0WmL6J6M9yCw5klGSb.w7OWbfmeCgUopj4vavKfHXhEPvO3mA9OEqW6 TBxRqkv2miAFi_ILYLFOIBw2o_XuTD4wzkPvc3B8pREfB75HOZaBMwpbgVirW2aXsT.M37cmJdNg G.7Adn.s6sXooyaEFr_RPhN_U.QQ2RzLgXS77DYakeeJP53AZkd4jEAqB4Ab8cZKrgL2qxq2ntij eAy3Eu464gF51ePGKAz.TowoNWHoMfVykRPDFawdVf0k7kgqSKYuGsCQNO1cEJH2sM4apaIPHyz. uAXukqWwl2_pHmmXn0JriszAi37.i99nFjw23.te4WFMv2L77loq.2_sJnI8FSBRUMSdJNTNSHri 6yvXa9ZLLfXwzUWFoEwBhaTdZaQd_OYqVm06pbgbi8PN9Kbx7V8JeVZO.LlAqhvdcXt2UfVC4s00 jNhfkNkBam7J4TllbSPFbCp75bw6i12nmu7Z1U9l_3y8tXrQT43GCUY3AQpHcDrexN9P8fE04Afi quGuPKNaFDSE6.O0IsI9N5ODYvShQiz24RYf_4.qcAh5tdAR0nAkgHZJLz6h9k5cg52r9BwsCciA SBYvtMNwiIL1YqWB0azB4OnV0.4BDBgxLGOqrxgKYIU496nyj3HllDAWGT9fOrapqWBSxU8FVQNy UAsL5NaZXVmY7uoabyOFmX6v7c0D2tw0syCDAqmNBqQt8AmtcssBnP.PT4GBvMx9BUhCZRXGkiM5 Ea2ImKcsepEl42z3EQB33CwQ9qVINmHygbAEnwiiH0OstMIX8efdeKIiuoo4Luqe8xuZl7AX7WLU 3RvHD5LDYMXpdVh7bEaSZmPV5uHKc9KdfAthxyK1aaGmiQaNbOLzEHAT18yVJCLj9UdJp5pUElBz lYP..UeTjekBfrVU0hfZm0h5A6_f4tn4atWcWH8zy_CEjKjuMo0xflKFRkJzqisSmLMKLyMi0yFc r7HfrTbKwa_QgltAoSIXctF4QJ6N2yyfn2bdWLX3D8DjZXGQoJOcrPrx5fhsiPNA3efZ1G_k5BX_ FNGMJIPm1ygU5NPSf3UpxDzBw7tXve2ZTb03qAin87eFBwYCyaWy5L5mvxReTytDKkXjGz4RiQzZ 7897yZ8.C5in0.n7TZisXOaXP9tlYLZdnpBM3dzf06VijkR6gEGimnTMy0kHFjtiFLQfTIVedygQ dJbwNtRy9E7mLrUSqOQg4d1oQFQrH4XeRSEztHkQ_YTHQdHpmWTqNawdSrkLvz4TpkjojmAcr27z frWINeDbVlYhg8t6DQn9KBHUaP4tPFR7ZhbBM2gWTgQkFTp2ajWk6ZmHBGmuWFdRttXEgf.BVPdN 0Ft9kIXx.IsHT6eoIDJ2T0QcElszKulgzzm03BjkdHvmE1L_99v6TvYh5clDSu0odq.3Mou.hWXH NkR3OoenCDncgrsni.ww.kCbcjF_PGkqZt1CttawjQ4Kqiksa6W.xKW98SlKJc8dFwFxHFC9T67q jtE6VblyAUhd00.c2.HCPUxPnkdLSrlM_tHihthTlr3iyhQzhdouP6bAIS4aaJ5.ctiPR_WcIKXL h2slnuBAaWiRtYAhpR2JYv72lWGqC3lYZ.3sDTU_Tt8yS5TEJEqHolHIsVLkyaDP5bxXTHHGM6H3 TqB3AvT7jKwSqs2du9aVMSsLstRRLoYb8ysHlanb0hQ4MhUN604NFvh75IOBmGR.rJgxla.fV98y uNwXrkOMb2zyrcNCqZsPL7qTLcagF1mHtSlxEStDdm7lZ6deF.IfQXBktnVfNIoyyzJYbwFNDMgn YEeokf2pqVMmc6FBg0AUOhK77Y80LpY.vFvxJXzolCAxS2JOOdBwojTAYqLuUalOcRMQeerkSsVC fwc6KLqaldIPDGuwzOCLbqbR1wB7lIjmEbdUMhQRyFzZ_Ep7ztF_k4NlKNK1P_8jfbak7aP7rhs6 LCQYeGW_36QqV0hwdiE5DZxjVK5uREd2JWXPYok77jRr4hXWP50Bh228frsAletpGg6kJEzL4np5 pffShwTZZVrub2TCBioGFQXdjvE_jZ6IikrKdV2JyHY1.up92SsDShhB7hUe8 X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Wed, 24 Feb 2021 23:53:39 +0000 Received: by smtp409.mail.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 91f204b8deec7cb2386249da4c69927a; Wed, 24 Feb 2021 23:53:37 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: The out-of-swap killer makes poor choices From: Mark Millard In-Reply-To: Date: Wed, 24 Feb 2021 15:53:37 -0800 Cc: Alan Somers , FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <90EC4887-A29A-4829-B75B-1D88303791A4@yahoo.com> References: <1984125.0OzZcVfBr4@ravel> To: Konstantin Belousov X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4DmCRn2Dcqz4pHp X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.66.147:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.66.147:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 23:53:42 -0000 On 2021-Feb-24, at 11:59, Mark Millard wrote: > On 2021-Feb-24, at 10:36, Konstantin Belousov = wrote: >=20 >> On Wed, Feb 24, 2021 at 10:34:23AM -0700, Alan Somers wrote: >>> There's another silly problem that I didn't mention in my original = post. >>> The old rule of thumb is that the swap partition's size should be = twice as >>> large as the amount of RAM. However, that's no longer possible in = many >>> cases. The kernel imposes a hard limit of 64 GiB (on amd64 at = least) on >>> the usable size of any swap partition, and many servers now have far = more >>> than 64 GiB of RAM. So the advice needs to change with the times. = I don't >> I do not think so. The usable size of the swap is determined by the >> amount of swap metadata we pre-configure at boot time. Usually it is >> sized proportionally to the available physical memory, but you can >> override swap zones size manually with the knob. >=20 > There was a period of time when the 128 GiByte RAM ThreadRipper > had its previous 192 GiByte swap partition use rejected and I > had to split it into 3 64 GiByte ones. Later I saw a checkin that > was a correction to some calculation (vague memory) and I retried > having one 192 GiByte swap partition and it was again allowed. >=20 > The ability to dump to a swap partition when there was a > 64 GiByte limitation with 128 GiByte of RAM had implications > for the configuration. I actually arranged having a partition > that was only used for dump's potential use. That took some > rearrangement to form a large enough space, making other > tradeoffs to do so. >=20 >=20 > (I'm not sure if I can find the commit that lead to me switching > back to more than 64 GiByte for a swap file on the large memory > machine. I do not remember details any more.) The 64 GiByte size limit (as seen in my environment) was replaced in: = https://cgit.freebsd.org/src/commit/sys/vm/swap_pager.c?id=3D00fd73d2dabde= e2638203dd1145f007787f05be9 a.k.a.: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D363532 QUOTE author Doug Moore 2020-07-25 18:29:10 +0000 committer Doug Moore 2020-07-25 18:29:10 = +0000 . . . Fix an overflow bug in the blist allocator that needlessly capped max swap size by dividing a value, which was always a multiple of 64, by 64. Remove the code that reduced max swap size down to that cap. Eliminate the distinction between BLIST_BMAP_RADIX and BLIST_META_RADIX. Call them both BLIST_RADIX. Make improvments to the blist self-test code to silence compiler warnings and to test larger blists. Reported by: jmallett Reviewed by: alc Discussed with: kib Tested by: pho Differential Revision:=09 https://reviews.freebsd.org/D25736 Notes Notes: svn path=3D/head/; revision=3D363532 END QUOTE Evidence sequence leading me there: Establish a large swap partition on a device with an old snapshot of my ThreadRipper environment, resulting in: # gpart show -pl nvd1 =3D> 40 937703008 nvd1 GPT (447G) 40 1024 nvd1p1 FBSDFSSDboot (512K) 1064 746586112 nvd1p2 FBSDFSSDroot (356G) 746587176 191115872 nvd1p3 FBSDFSSDswap (91G) I got a kernel from the ci.freebsd.org artifacts and put it in place on the old snapshot of my ThreadRipper environment (that no longer could even boot --ACPI incompatibilities), so updating the old failing kernel but leaving the rest unchanged: # uname -apKU FreeBSD FBSDFSSD 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r358314: Tue Feb = 25 18:08:20 UTC 2020 = root@FreeBSD-head-amd64-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/amd64.a= md64/sys/GENERIC amd64 amd64 1300081 1300037 So: old head (13) environment booted on the 128 GiByte ThreadRipper: =46rom /var/log/messages: WARNING: reducing swap size to maximum of 65536MB per unit # swapinfo Device 1K-blocks Used Avail Capacity /dev/gpt/FBSDFSSDswap 67108864 0 67108864 0% The code that produced the message and limited the size was in sys/vm/swap_pager.c back in that time frame: static void swaponsomething(struct vnode *vp, void *id, u_long nblks, sw_strategy_t *strategy, sw_close_t *close, dev_t dev, int flags) { struct swdevt *sp, *tsp; swblk_t dvbase; u_long mblocks; =20 /* * nblks is in DEV_BSIZE'd chunks, convert to PAGE_SIZE'd = chunks. * First chop nblks off to page-align it, then convert. * * sw->sw_nblks is in page-sized chunks now too. */ nblks &=3D ~(ctodb(1) - 1); nblks =3D dbtoc(nblks); =20 /* * If we go beyond this, we get overflows in the radix * tree bitmap code. */ mblocks =3D 0x40000000 / BLIST_META_RADIX; if (nblks > mblocks) { printf( "WARNING: reducing swap size to maximum of %luMB per unit\n", mblocks / 1024 / 1024 * PAGE_SIZE); nblks =3D mblocks; } . . . Then I used blame to find the fix in git via looking at: https://cgit.freebsd.org/src/blame/sys/vm/swap_pager.c >> know what the best size would be for a modern server, but I would = guess >>> that it must be at least several times the RSS of your largest = process, and >>> also at least one tenth of RAM (for use as a dump device with = compressed >>> core dumps). =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Wed Feb 24 23:55:59 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 44F0754D888 for ; Wed, 24 Feb 2021 23:55:59 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmCVR1TDyz4pG3; Wed, 24 Feb 2021 23:55:59 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f42.google.com with SMTP id c16so4034223otp.0; Wed, 24 Feb 2021 15:55:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cj3VZ3pVoqIO+Fvrh7C9l44ieHnc+14znhk/bQWQUYQ=; b=ufIUTIALsC3iYS0U03nADUSDPeso3pFe7PTUIS08760GpunRBg7ee/0jxgeKqmpuxN CR+GBDU7ZnrK5adE4gS+rx0FhOW8+2iNcKtbIMp19uIVZEY4fsC86dJuaJYAgdKxdu0k 6+roAoGaQZANjdPLQ8gJmjOVmGqs9pyqWDTxLoDERXIc6M70azxOS1LLtsmMU08xEK+O a6UE7qRrwRX8xQkMcf23O46DHqoAQ45R7x8SsEMla8cc9p4jJ9boFshidMRZUiTR9KfF jZcMC6WxNEmokieWaGcxYEt7cBvAHFTyBbOwcb+YgYf9njf4Db+391TAaOQ7Gt8816pw 81MA== X-Gm-Message-State: AOAM532kzgZsQ4drCMMiXGqP1wFtmlWZyPnzZywChiOsUxQZ/lY3HS91 5r5TFnVOILMJYXqZrZG6KGepZOwtpaMTEoFUri4= X-Google-Smtp-Source: ABdhPJx6e/MZ6x654tlXL9cubXA/rXZfw7zrO/oRAmErXpoZubwyc+YoxxeSRUzpHZdg3WNLm0+QOeEwO+Q7+1/emsM= X-Received: by 2002:a9d:3642:: with SMTP id w60mr64287otb.18.1614210958000; Wed, 24 Feb 2021 15:55:58 -0800 (PST) MIME-Version: 1.0 References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> In-Reply-To: From: Alan Somers Date: Wed, 24 Feb 2021 16:55:46 -0700 Message-ID: Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko To: Konstantin Belousov Cc: Ravi Pokala , "freebsd-hackers@freebsd.org" X-Rspamd-Queue-Id: 4DmCVR1TDyz4pG3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 23:55:59 -0000 On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov wrote: > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > Hi folks, > > > > A colleague and I both independently observed `sysctl -a' appear to hang > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C > didn't kill it. We could still establish a new terminal session to the > node, via SSH or serial console, and we were able to see that it was > actually spinning, not hung, and was consuming an entire CPU. > > > > We eventually determined that it was specifically `sysctl > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > spinned if nvdimm.ko was loaded. It was not necessary to access the device > node associated with the NVDIMM; merely having the module loaded was > sufficient. > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who > uses it can at least confirm my findings on this. Help in debugging would > be even more appreciated. > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this > could be quite large. It could be busy dumping page tables. > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs all other sysctls combined, and tends to grow with time. Would it be possible to hide it from sysctl -a's output? I think there are other sysctls like that, that are treated as opaque binary values. I once had to fix a bug in py37-salt that caused a common operation to take _6_hours_ as opposed to < 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some O(n^2) string processing. From owner-freebsd-hackers@freebsd.org Thu Feb 25 00:26:01 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0CAC954E730 for ; Thu, 25 Feb 2021 00:26:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmD9458R7z4rY5; Thu, 25 Feb 2021 00:26:00 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11P0PrxP021161 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 25 Feb 2021 02:25:56 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11P0PrxP021161 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11P0Prwd021160; Thu, 25 Feb 2021 02:25:53 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 25 Feb 2021 02:25:53 +0200 From: Konstantin Belousov To: Alan Somers Cc: Ravi Pokala , "freebsd-hackers@freebsd.org" Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Message-ID: References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DmD9458R7z4rY5 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 00:26:01 -0000 On Wed, Feb 24, 2021 at 04:55:46PM -0700, Alan Somers wrote: > On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov > wrote: > > > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > > Hi folks, > > > > > > A colleague and I both independently observed `sysctl -a' appear to hang > > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C > > didn't kill it. We could still establish a new terminal session to the > > node, via SSH or serial console, and we were able to see that it was > > actually spinning, not hung, and was consuming an entire CPU. > > > > > > We eventually determined that it was specifically `sysctl > > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > > spinned if nvdimm.ko was loaded. It was not necessary to access the device > > node associated with the NVDIMM; merely having the module loaded was > > sufficient. > > > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who > > uses it can at least confirm my findings on this. Help in debugging would > > be even more appreciated. > > > > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this > > could be quite large. It could be busy dumping page tables. > > > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). > > > > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs all > other sysctls combined, and tends to grow with time. Would it be possible > to hide it from sysctl -a's output? I think there are other sysctls like > that, that are treated as opaque binary values. I once had to fix a bug in > py37-salt that caused a common operation to take _6_hours_ as opposed to < > 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some > O(n^2) string processing. jhb already marked this sysctl ask CTLFLAG_SKIP, several months ago. The change was not merged back to 12. From owner-freebsd-hackers@freebsd.org Thu Feb 25 00:27:54 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7A29A54ED41 for ; Thu, 25 Feb 2021 00:27:54 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmDCG30lbz4rlQ; Thu, 25 Feb 2021 00:27:54 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f53.google.com with SMTP id b16so4106894otq.1; Wed, 24 Feb 2021 16:27:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NMKei3kFXb5poPz53EJcYfntqYmtUsdXaLP6Rh0Z0lg=; b=XHgM7mmZDWEvV5EDVbLiZd4mVnNjiaFfAhN1rF0P5vMYlkbZDYCEm1bXNNfqPgKkYb cC/1lgdMBuB3uprqoC3Bkqmklf88+t0Mx2eOE9rvvfyQl8/hkTwvAe0xNf5rzVNwuzOP Fs3qAH02fayTF7SqwsFE8ACg5tf2AywdJxAbeYuf1yYMzut7H8mqmEV1sdvNGY6CXgoC j/Qi3SEtM8jOvlL6vmi4gqTJTbgT7B47zUd8TnlzRfhfNMLnv/Oil9zs7FqwfSBuoIjq CnK6YzS+GTsQYeyBcHmc9svFy+Ubd+xIA171JTqTfARpU81aqb6//tkgSTwMjkLmbSBI wU8w== X-Gm-Message-State: AOAM533Yu3zhbRq+ffhP0tAUQCxDQS6xSUYpSU2Ve0L+m2NKikkpjCKT T3LRb+18ljIBbLl5IcHoClEqoIBoOhS3THMAjDo= X-Google-Smtp-Source: ABdhPJxrpX+JViaM/MxGTEwcpleBgUCAdA87lowznqWVmNi4Az05fHhtMg3+Er8BoXhVD6WTjfERH3y0IfZbGphH+TU= X-Received: by 2002:a9d:3642:: with SMTP id w60mr160352otb.18.1614212873367; Wed, 24 Feb 2021 16:27:53 -0800 (PST) MIME-Version: 1.0 References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> In-Reply-To: From: Alan Somers Date: Wed, 24 Feb 2021 17:27:42 -0700 Message-ID: Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko To: Konstantin Belousov Cc: Ravi Pokala , "freebsd-hackers@freebsd.org" X-Rspamd-Queue-Id: 4DmDCG30lbz4rlQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 00:27:54 -0000 On Wed, Feb 24, 2021 at 5:26 PM Konstantin Belousov wrote: > On Wed, Feb 24, 2021 at 04:55:46PM -0700, Alan Somers wrote: > > On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov > > > wrote: > > > > > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > > > Hi folks, > > > > > > > > A colleague and I both independently observed `sysctl -a' appear to > hang > > > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, > and ^C > > > didn't kill it. We could still establish a new terminal session to the > > > node, via SSH or serial console, and we were able to see that it was > > > actually spinning, not hung, and was consuming an entire CPU. > > > > > > > > We eventually determined that it was specifically `sysctl > > > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > > > spinned if nvdimm.ko was loaded. It was not necessary to access the > device > > > node associated with the NVDIMM; merely having the module loaded was > > > sufficient. > > > > > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone > who > > > uses it can at least confirm my findings on this. Help in debugging > would > > > be even more appreciated. > > > > > > > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and > this > > > could be quite large. It could be busy dumping page tables. > > > > > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over > it). > > > > > > > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs > all > > other sysctls combined, and tends to grow with time. Would it be > possible > > to hide it from sysctl -a's output? I think there are other sysctls like > > that, that are treated as opaque binary values. I once had to fix a bug > in > > py37-salt that caused a common operation to take _6_hours_ as opposed to > < > > 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some > > O(n^2) string processing. > > jhb already marked this sysctl ask CTLFLAG_SKIP, several months ago. > The change was not merged back to 12. > Ahh, good. From owner-freebsd-hackers@freebsd.org Thu Feb 25 02:06:09 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6F6EA552878 for ; Thu, 25 Feb 2021 02:06:09 +0000 (UTC) (envelope-from rpokala@panasas.com) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2054.outbound.protection.outlook.com [40.107.236.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmGNd1ck7z3GCh; Thu, 25 Feb 2021 02:06:08 +0000 (UTC) (envelope-from rpokala@panasas.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MpDmdPWmTkOY0FGy/erZ2A5o5m9sVGu4o8GXHKssOdK21BOad8kD4NsjZZTGbPLDyG1tNymk9Y06HS7p9NI3SlWoUFG+bbwFr19MZ6Iv2Vhufe6wpr+pe6S0Ja6GuGm9Hjix4Yr3UJA4yBl09+rXZncxhNXz4AQSsAO6tSY6CjlhV5CDR6Sbgzu6Hc9wi8Z7CZeiKi66sr6anU+FFMFKEMukq1T3mShgPWHImxaVhDiuUD2qcVspdqxsnhWjcqkZhet7WkKvaWgQkEj2YOVprZIb8UOFo22ncmjgYN4xcCWnnM2HfDSVGinN69zEWZSW3zYkJ2FzIuYSiPj695nCyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lhz4TgL+AMLCujPFE7wDiNK9Gg8ESZOSZtk3XYrMNYo=; b=VZq1ec9LqWLt0jmArQqObldq5fWI8KYO4CmbFtGVPcddX5mUlT015nwek4a5vyhL6aa2dV/2NoJShVjxnd9FEEdfPnUKPugwF5E5ht3a8QPahgtMEKNeeH7plHilSKmTmwUbr4x7x1o+ag7ka9Xs1hLecJA/F7FOEWzGAQIbkuwK2f03GnI2vLs/UenJmdPWu6hmTaf7iMqBw7nmosdxTnu9i1ykB8TfZkJUFoyXQH2K2f51rmC3SIbRZsTPd3X00xGKUd7GbAHmV3Lg1T+PHb8ruh7/87EMi4oj5Rz1eJ+HfpAX+86koMLtwHn2Nfp83wsAHiefPYokBcii75OvXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=panasas.com; dmarc=pass action=none header.from=panasas.com; dkim=pass header.d=panasas.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=panasas.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lhz4TgL+AMLCujPFE7wDiNK9Gg8ESZOSZtk3XYrMNYo=; b=WS5mrR2HkK4ZQCY0UGrJ8m7Wcc8H6c9fmdTHe7D+KiazA95X+jeFL3ayVFP5nvw6S2geiEmtdKb9mBAoT3ciN/gBgiyIZS9SlcW/sDuhHzjHLqLFwMDFSW11g3gx2TAX+SGzLjk0ya1rCaipy703/rBsslQeDLcL6ALDAQglSeE= Received: from BYAPR08MB4966.namprd08.prod.outlook.com (2603:10b6:a03:66::32) by BY3PR08MB7123.namprd08.prod.outlook.com (2603:10b6:a03:362::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3868.27; Thu, 25 Feb 2021 02:06:05 +0000 Received: from BYAPR08MB4966.namprd08.prod.outlook.com ([fe80::fd82:ae89:c5dd:a3a4]) by BYAPR08MB4966.namprd08.prod.outlook.com ([fe80::fd82:ae89:c5dd:a3a4%7]) with mapi id 15.20.3868.032; Thu, 25 Feb 2021 02:06:05 +0000 From: "Pokala, Ravi" To: Alan Somers , Konstantin Belousov CC: "freebsd-hackers@freebsd.org" Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Thread-Topic: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Thread-Index: AQHXCxrEm9ne0qx3GkGxLYQsTySE0Q== Date: Thu, 25 Feb 2021 02:06:04 +0000 Message-ID: <983A98FE-9D99-4E6F-9F5F-7719EE258EB6@panasas.com> References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/16.45.21011103 x-originating-ip: [98.42.164.217] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: efc7f6ed-7bd9-4f32-a4e5-08d8d931e8c8 x-ms-traffictypediagnostic: BY3PR08MB7123: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: MGpmGFaIZnfT2+6C2rKOLhOrSG3gw8CG3Zg98fIZxNUMYeyONmR7mFFbbPAX5g1IS3KaBxp+SnRHHfBas0kM2Nw0DihARkYupRsl7+FcmauFmV5MolfbB/TR5NibQSlF6DYlzK4SlC7+CltbJMTOCP8St3Cejp+96niIWGT35omsKRJANBEtpKxvfl6RnV0aZ02SeR2gLwXbp/3c7G2Q996Lw2y2ENOCCdzLW/vCOeFTIKef3JPerrXksfiok+Jx5cIJtQZB7VrOKwglUnwlkGg8FiQjlZn9nLMyAqObFw3R3A7YTengRZMM/NvMJ/KbfE6dQTYtpn27fQBFX1gzK2bXiZuXbWMNbTATkCxhqJsdh08v4xD7JZC0rCgJdaWJlD86KmYRcWqFhVwfVciSmT3MU/tqtjvf2INAwzJdgyVCYlhreaqfA/gjnWeflKjRrE16X14JWjj2DWMQ1EZRBYLmZJxqepN6/1eqNO/ZducPaiH0GkFuVI7zf0CxuKN3PnDO5NqaEb+ar+nIgI+eTHUQ5L60n4K7pNiDK0aLyalHv1HbHdqfQrEYkWNC+XfG9vu8EqtOEDgmSsj3tspTkcmz7fPIFqR9gJvegr1veYY= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR08MB4966.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(39840400004)(396003)(366004)(136003)(346002)(376002)(2616005)(8936002)(8676002)(86362001)(33656002)(6506007)(316002)(110136005)(6512007)(186003)(53546011)(6486002)(478600001)(64756008)(66556008)(66446008)(66476007)(71200400001)(2906002)(36756003)(4326008)(5660300002)(26005)(83380400001)(66946007)(76116006)(81973001)(45980500001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?utf-8?B?d1kwbndKcGZ5dFcvZk1Rb25oVTBQeERtbGZ5UjB1M2p3RjBaSUpJbERsNWFv?= =?utf-8?B?MDg4VXkyTUNZY1VoRERKMkhiNE8wVE1VanRnL2hPTGhSSk03T2NON2JZQnJh?= =?utf-8?B?c2FMcXg4OENzZDl1WFAxZkFqaFYwY1Rya1EwOVNuR0krQ05sSDYyVE9GdWsr?= =?utf-8?B?RFh6bUN1K0h6MGtMRTNZZUNLRzFHNWttZ3BIdlZCVU1CYlpqWTRGOWFTakZG?= =?utf-8?B?aFhlTUtMUGtnRUxGeDBmY2NxRkZiU05iR25EKzErNDhIWWxFMVFOazJ0Rkpy?= =?utf-8?B?TGt2aEZmZkdUcEpLUjNDWGZzSFJURCtlUFRSd0lqUVVCN2NQYXBOMmVOSUF0?= =?utf-8?B?VXljRUVEZnRlN1czUUJZbkFJNkZKV0dVVlR2Zm5Ya0wwRlJWd01NTGNScEZL?= =?utf-8?B?elErd3IveVh6SDBnL2t6MXNaQ2VueXBBOHF4Q24rS1ZiMXhlL1g0S1pnRVVy?= =?utf-8?B?dys1Z2pZSERMZUp1UzJrTEtSWXlpQStSbi9XVG1ORi9tMmlXd3JadW1jZlhX?= =?utf-8?B?SCtDUTk5M3ozcXIrM1kvK3dOTStwa3JIbnJkMC9ydWNIeEFYQWlDdDl0Q2Zr?= =?utf-8?B?N3dFQUJpeExMS3JGYzlSV2wvclRRWGRSQ2c3MU5icU5PSHp3UkJnWFVvVFFC?= =?utf-8?B?RGlMSXpRZUtvMTZTQlJzN1RITVhDeVd4QUNTQ3g2YVAybGtJVDI3aHlxSEd4?= =?utf-8?B?TlYzcUE1OVhKL3ZWYXVwMmV5NzFMTVNvUTFrcjhvUjIyWHZWZUJTVXp3UWs4?= =?utf-8?B?TGN1RGJHVjFFdEo4b0tyVXZxVXM1Z1lXeE5YRXY4cisyUEc3cDhTWGtzTjVy?= =?utf-8?B?akhaVHhHUDgrMWh6a1JJMFBUVnpMVmdwYUZta0g3UG1oMDVDalVqOHZ5RzR4?= =?utf-8?B?RnZncDdVYTZ2bHFESzN5QVByeVU0Q2FnYXhFNmRWT0pwWmUraFBIZVJsc3NS?= =?utf-8?B?SFFkcWR2aGpCUGFrV1F1dWZGYlI3QVVNV2pWSmRrMlZIRFFoN1VTY2Z2YWw1?= =?utf-8?B?UmhIZDRBT1JSdG1HYzZxK1F2YU51d0J0YjNtWVRGTEwzQVZNTVJSaVM0VklJ?= =?utf-8?B?UnpBNVdVaUo2d2hjMkhvV3VYTU96ZmgzSjYxZGJEY1l0TDZiQ2xWNVN5Q1VV?= =?utf-8?B?SndhTXZlY2dMdmNIcUY2VnUwOGp3dGQ5UGJZTlV2NFZ6NXk2aVRKTFFVdDBT?= =?utf-8?B?UE5GVUZ3UWN5Ti9JL01xNVJLUW5nRzhoOGZMN3N2SkxMQkVnK1J5R05YMFd1?= =?utf-8?B?Q2ZLWDhYcGpqM08zMC9PL01MUU1YSEN0S0hmc2tnVHFZQVNhS2UybDR6ZWd5?= =?utf-8?B?bFMvanQ4azRXVkQxUER6VVNQdy9EMUNCSDVPRVFEL1AyKys2OVQ2ek1jakhv?= =?utf-8?B?bWZTaG5WTjJuWFMyTmdjWTlOR0d0b282dW9ONmlKOVRIVUJBNjVIVzcyNFM2?= =?utf-8?B?KzBDRlpJMU45L0NHaXRxelV4VWJ1R2I1LzZ3aktpM3ladGNkK3BiVHQwUW0x?= =?utf-8?B?VURBcUMwSmsyU21zNnI5VzlSaXNRSS8vWStpa2xrUjRUNmN4NmF0cEJyN0Ur?= =?utf-8?B?OVZRWmlrZGc5WXV4WGNkNUZka1ZacDlnd3JtcmNOdm1JUzRNVzlvNWFXSlNq?= =?utf-8?B?NnR1WWRxNGl4NC9GUTZ2UFF0dWNOTzZFdmJzaU9uYjlOSTVFUzJaMURRbGJI?= =?utf-8?B?TUhhQmhHcmNDNjJMcnhwVXBMa2FCZng1aXRvby8rZG52MHoyMC9uODdzckha?= =?utf-8?B?VUtXTEFPSjgvc0p5aS9tT2ZES1hXZXNTWVhGYlRFVVlMTkMyQU82TWE0NWVq?= =?utf-8?B?Z3RzWFN6Zk11VXMyRFpsdz09?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="utf-8" Content-ID: <7DD6030668F02741AD123A8436CA7A6C@namprd08.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR08MB4966.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: efc7f6ed-7bd9-4f32-a4e5-08d8d931e8c8 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Feb 2021 02:06:04.9076 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 30TCsNMrQOQYqEcRXkXT+3Iz+tErOOaJTHjJfRE/DLggGqjcmof+nF0XJmfoCjXByTRgapLTn7bmAuDf3MQ90Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR08MB7123 X-Rspamd-Queue-Id: 4DmGNd1ck7z3GCh X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 02:06:09 -0000 DQoNCu+7vy0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBBbGFuIFNvbWVycyA8YXNv bWVyc0BmcmVlYnNkLm9yZz4NCkRhdGU6IDIwMjEtMDItMjQsIFdlZG5lc2RheSBhdCAxNjoyNw0K VG86IEtvbnN0YW50aW4gQmVsb3Vzb3YgPGtvc3Rpa2JlbEBnbWFpbC5jb20+DQpDYzogUmF2aSBQ b2thbGEgPHJwb2thbGFAZnJlZWJzZC5vcmc+LCAiZnJlZWJzZC1oYWNrZXJzQGZyZWVic2Qub3Jn IiA8ZnJlZWJzZC1oYWNrZXJzQGZyZWVic2Qub3JnPg0KU3ViamVjdDogUmU6IGBzeXNjdGwgdm0u cG1hcC5rZXJuZWxfbWFwcycgc3BpbnMgb24gMTIuMi1SRUxFQVNFLXAzIHcvIG52ZGltbS5rbw0K DQogICAgT24gV2VkLCBGZWIgMjQsIDIwMjEgYXQgNToyNiBQTSBLb25zdGFudGluIEJlbG91c292 IDxrb3N0aWtiZWxAZ21haWwuY29tPiB3cm90ZToNCg0KDQogICAgT24gV2VkLCBGZWIgMjQsIDIw MjEgYXQgMDQ6NTU6NDZQTSAtMDcwMCwgQWxhbiBTb21lcnMgd3JvdGU6DQogICAgPiBPbiBXZWQs IEZlYiAyNCwgMjAyMSBhdCA0OjQ5IFBNIEtvbnN0YW50aW4gQmVsb3Vzb3YgPGtvc3Rpa2JlbEBn bWFpbC5jb20+DQogICAgPiB3cm90ZToNCiAgICA+IA0KICAgID4gPiBPbiBXZWQsIEZlYiAyNCwg MjAyMSBhdCAwMzozNzoxMlBNIC0wODAwLCBSYXZpIFBva2FsYSB3cm90ZToNCiAgICA+ID4gPiBI aSBmb2xrcywNCiAgICA+ID4gPg0KICAgID4gPiA+IEEgY29sbGVhZ3VlIGFuZCBJIGJvdGggaW5k ZXBlbmRlbnRseSBvYnNlcnZlZCBgc3lzY3RsIC1hJyBhcHBlYXIgdG8gaGFuZw0KICAgID4gPiBv biBub2RlcyBydW5uaW5nIEZyZWVCU0QgMTIuMi1SRUxFQVNFLXAzOyBpdCBkaWRuJ3QgZW1pdCBh bnkgb3V0cHV0LCBhbmQgXkMNCiAgICA+ID4gZGlkbid0IGtpbGwgaXQuIFdlIGNvdWxkIHN0aWxs IGVzdGFibGlzaCBhIG5ldyB0ZXJtaW5hbCBzZXNzaW9uIHRvIHRoZQ0KICAgID4gPiBub2RlLCB2 aWEgU1NIIG9yIHNlcmlhbCBjb25zb2xlLCBhbmQgd2Ugd2VyZSBhYmxlIHRvIHNlZSB0aGF0IGl0 IHdhcw0KICAgID4gPiBhY3R1YWxseSBzcGlubmluZywgbm90IGh1bmcsIGFuZCB3YXMgY29uc3Vt aW5nIGFuIGVudGlyZSBDUFUuDQogICAgPiA+ID4NCiAgICA+ID4gPiBXZSBldmVudHVhbGx5IGRl dGVybWluZWQgdGhhdCBpdCB3YXMgc3BlY2lmaWNhbGx5IGBzeXNjdGwNCiAgICA+ID4gdm0ucG1h cC5rZXJuZWxfbWFwcycgd2hpY2ggd2FzIHNwaW5uaW5nLCBhbmQgc3Vic2VxdWVudGx5IHRoYXQg aXQgb25seQ0KICAgID4gPiBzcGlubmVkIGlmIG52ZGltbS5rbyB3YXMgbG9hZGVkLiBJdCB3YXMg bm90IG5lY2Vzc2FyeSB0byBhY2Nlc3MgdGhlIGRldmljZQ0KICAgID4gPiBub2RlIGFzc29jaWF0 ZWQgd2l0aCB0aGUgTlZESU1NOyBtZXJlbHkgaGF2aW5nIHRoZSBtb2R1bGUgbG9hZGVkIHdhcw0K ICAgID4gPiBzdWZmaWNpZW50Lg0KICAgID4gPiA+DQogICAgPiA+ID4gSSBrbm93IG52ZGltbSg0 KSBpc24ndCB0ZXJyaWJseSB3aWRlbHkgdXNlZCwgYnV0IGhvcGVmdWxseSBzb21lb25lIHdobw0K ICAgID4gPiB1c2VzIGl0IGNhbiBhdCBsZWFzdCBjb25maXJtIG15IGZpbmRpbmdzIG9uIHRoaXMu IEhlbHAgaW4gZGVidWdnaW5nIHdvdWxkDQogICAgPiA+IGJlIGV2ZW4gbW9yZSBhcHByZWNpYXRl ZC4NCiAgICA+ID4gPg0KICAgID4gPg0KICAgID4gPiBIb3cgbGFyZ2UgeW91ciBudmRpbW1zIGFy ZT8gIFRoZWlyJyBTUEFzIGFyZSBtYXBwZWQgaW50byBLVkEgZnVsbHkgYW5kIHRoaXMNCiAgICA+ ID4gY291bGQgYmUgcXVpdGUgbGFyZ2UuICBJdCBjb3VsZCBiZSBidXN5IGR1bXBpbmcgcGFnZSB0 YWJsZXMuDQoNCk9uIHRoZXNlIG5vZGVzLCAxNkdCLg0KDQogICAgPiA+IFRyeSB0byBza2lwIGxh cmdlIG1hcCBpbiBwbWFwLmM6c3lzY3RsX2ttYXBzKCkgKGp1c3QgaW5jcmVtZW50IGkgb3ZlciBp dCkuDQoNClRoYW5rcyEgVGhpcyB3b3JrZWQgZm9yIG1lOg0KDQp8ICAJCWNhc2UgTE1TUE1MNEk6 DQp8IC0JCQlzYnVmX3ByaW50ZihzYiwgIlxuTGFyZ2UgbWFwOlxuIik7DQp8IC0JCQlicmVhazsN CnwgKwkJCXNidWZfcHJpbnRmKHNiLCAiXG5MYXJnZSBtYXA6IFNLSVBQSU5HXG4iKTsNCnwgKwkJ CWNvbnRpbnVlOw0KfCAgCQl9DQoNCiAgICA+IFNwZWFraW5nIG9mIHZtLnBtYXAua2VybmVsX21h cHMsIHRoYXQgdGhpbmcgaXMgaHVnZS4gIEl0IGVhc2lseSBkd2FyZnMgYWxsDQogICAgPiBvdGhl ciBzeXNjdGxzIGNvbWJpbmVkLCBhbmQgdGVuZHMgdG8gZ3JvdyB3aXRoIHRpbWUuICBXb3VsZCBp dCBiZSBwb3NzaWJsZQ0KICAgID4gdG8gaGlkZSBpdCBmcm9tIHN5c2N0bCAtYSdzIG91dHB1dD8g IEkgdGhpbmsgdGhlcmUgYXJlIG90aGVyIHN5c2N0bHMgbGlrZQ0KICAgID4gdGhhdCwgdGhhdCBh cmUgdHJlYXRlZCBhcyBvcGFxdWUgYmluYXJ5IHZhbHVlcy4gIEkgb25jZSBoYWQgdG8gZml4IGEg YnVnIGluDQogICAgPiBweTM3LXNhbHQgdGhhdCBjYXVzZWQgYSBjb21tb24gb3BlcmF0aW9uIHRv IHRha2UgXzZfaG91cnNfIGFzIG9wcG9zZWQgdG8gPA0KICAgID4gMSBtaW51dGUgYmVjYXVzZSBv ZiBhIGh1Z2Ugdm0ucG1hcC5rZXJuZWxfbWFwcyB2YWx1ZSwgY291cGxlZCB3aXRoIHNvbWUNCiAg ICA+IE8obl4yKSBzdHJpbmcgcHJvY2Vzc2luZy4NCg0KV2l0aCBudmRpbW0ua28gdW5sb2FkZWQg b24gdGhlc2Ugbm9kZXMsIGl0J3MgYXJvdW5kIDMwMDAgbGluZXMhIE9fTw0KDQpJdCBvY2N1cnJl ZCB0byBtZSB0aGF0IGl0IG1pZ2h0IGV2ZW50dWFsbHkgZmluaXNoLCBzbyBJIHN0YXJ0ZWQgaXQg bGFzdCBuaWdodCwgYW5kIGl0IGhhZG4ndCBjb21wbGV0ZWQgb3Zlcm5pZ2h0Lg0KDQogICAgamhi IGFscmVhZHkgbWFya2VkIHRoaXMgc3lzY3RsIGFzayBDVExGTEFHX1NLSVAsIHNldmVyYWwgbW9u dGhzIGFnby4NCiAgICBUaGUgY2hhbmdlIHdhcyBub3QgbWVyZ2VkIGJhY2sgdG8gMTIuDQoNCk5p Y2UhDQoNCkZvdW5kIGl0OiByMzY4NzY4ICgxZGNlN2Q5ZTdlZWYpDQoNClRoYXQncyBwcm9iYWJs eSBhIGNsZWFuZXIgZml4IHRoYW4gdGhlIHBhdGNoIGFib3ZlOyBJJ2xsIHNlZSBhYm91dCBhcHBs eWluZyB0aGF0IGluIG15IHRyZWUuDQoNClRoYW5rcywgYWxsIQ0KDQotUmF2aSAocnBva2FsYUAp DQoNCiAgICBBaGgsIGdvb2QuIA0KDQoNCg0K From owner-freebsd-hackers@freebsd.org Thu Feb 25 04:38:39 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A209555FD0 for ; Thu, 25 Feb 2021 04:38:39 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmKmZ4nGFz3PVj for ; Thu, 25 Feb 2021 04:38:38 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: by mail-ej1-x632.google.com with SMTP id w1so6579032ejf.11 for ; Wed, 24 Feb 2021 20:38:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=yBmBem/so8lEw88qbUitkdESVAqQMgvs3lO+uea7UA0=; b=ljb+jOlgHuZDwyFdHTuB4NDV+BEmLwAR5aOxPoJCyIDd4Xg0ul+6UUuChPk933Vu+Y fCyBeQYe8kldZePBVUZQYI01RsXreZOkqsnAy4jJrRYV1qxtRrk/AcGxyzuLXxAaTkzF lRjtnk3NTTCuEqlD0LooeehyRii8kqJoAuFxQfBHZugUBYnZcEdHI4IM1dNvqoIxux1w 3Az05CwBzmuXb7hB5com/18zO9GBjIdh4nHQzyIZY4artHDaG53ChzQS4W0rDzfhtknt 4B2DIg1cA9UyvYkMf477IGEIeFVa9M8OS7A8ITOeu16ZMF9Y8y/5jJuTW+0WPhypHqf6 lFZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=yBmBem/so8lEw88qbUitkdESVAqQMgvs3lO+uea7UA0=; b=KT0ilyrfvR21OGjs03e0JDUyUzpKZj0MxBUo4jaFj/HEnF1p52Brg7zziyCzJnkl0q CWOACDpwevV8JeuL8CSvFf6TmS60IBnq6ZxLPyQNd2tmiCnHg1Nqd5v9w0cX30L4gTPP G6DbpmQNe+dpQbbJPf9JDmvNd6KbHgrSoy25HHfYjkQAEPrHBAflLXBljeUhiQCe8Gch uSZ++IqG8bSmLK/bswoSb9WSRmIqYMVu+7Cuz0G8+3Qt05vWXJtf7XX6z9tQKojAuNA8 8xQGt57W8b626EvhQAKgjcj3rerjS7UsWDNtGvRyCfzGqR+TsmPOhxa37Xa0aul7cChL iX7w== X-Gm-Message-State: AOAM531NkMt3GV1u51obYru9W1jBsvfOgJPx97mg0SNuH6xlO+D8fGl1 YHvxhmt7I3b1BlZW0q+Z6Vyzm+UyVneLilbhu7M= X-Google-Smtp-Source: ABdhPJzMq6P9/74fvIuyUQLiY2d5hn+ddtQT+LA96rkI0hVHUHjgNe/JrLmMwG0as+wID5a+qjBbBSyW9AgQMSxiPEs= X-Received: by 2002:a17:906:3916:: with SMTP id f22mr922064eje.328.1614227917472; Wed, 24 Feb 2021 20:38:37 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a54:3148:0:0:0:0:0 with HTTP; Wed, 24 Feb 2021 20:38:36 -0800 (PST) From: grarpamp Date: Wed, 24 Feb 2021 23:38:36 -0500 Message-ID: Subject: CA's TLS Certificate Bundle in base = BAD To: freebsd-security@freebsd.org Cc: freebsd-questions@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DmKmZ4nGFz3PVj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ljb+jOlg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of grarpamp@gmail.com designates 2a00:1450:4864:20::632 as permitted sender) smtp.mailfrom=grarpamp@gmail.com X-Spamd-Result: default: False [-2.02 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::632:from]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.98)[0.984]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::632:from:127.0.2.255]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::632:from]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Mailman-Approved-At: Thu, 25 Feb 2021 18:23:24 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 04:38:39 -0000 FYI... Third party CA's are an untrusted automagical nightmare of global and local MITM risk... - CA's issuer gone wrong... Govt, Corp, Bribe, Rogue, Court, War, Force Majeure, Crime, Hack, Spies, Lulz, etc. - CA's store bundler gone wrong... Mozilla, Microsoft, Apple, BSD, etc in same ways above. - Undetected stolen unrevoked unchecked CA's, intermediates, server keys, etc. - Total/targeted IP/DNS traffic user interception by agents, vpn's, proxies, tor, mitmproxy, sslstrip, etc. - Base asserting trust over all that, when reality none is due. There should be no non-FreeBSD.Org/Foundation CA's shipped in base. Its shipped pubkey fingerprint sets can bootstrap TLS infra pubkeys/prints off bsd keyserver, to then pubkey pin TLS fetch(1) / pkg(8) / git(1) to reach pkg ca_root_cert, git src ports repos, update, iso, etc. See curl(1) --pinned-pubkey, GPG, etc. https://www.zdnet.com/article/surveillance-firm-asks-mozilla-to-be-included-in-firefoxs-certificate-whitelist/ https://en.wikipedia.org/wiki/Edward_Snowden https://duckduckgo.com/?q=rogue+CA+root+certificate https://www.win.tue.nl/hashclash/rogue-ca/ Users should delete all those ~139 garbage CA's, only add in the ones they find they need during use, easily scripted and tooled, start with say the... - LetsEncrypt chain And force TLS pubkey fingerprint pin check on critical services. Search web for howtos. At minimum require user / install to ack before use... mv /etc/ssl/certs.shipped_disabled /etc/ssl/certs From owner-freebsd-hackers@freebsd.org Sat Feb 27 01:06:13 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DF64D555D8C for ; Sat, 27 Feb 2021 01:06:13 +0000 (UTC) (envelope-from hackers@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DnSyY66F6z3Qp3 for ; Sat, 27 Feb 2021 01:06:13 +0000 (UTC) (envelope-from hackers@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id D1AC1555AE0; Sat, 27 Feb 2021 01:06:13 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1738555D8B for ; Sat, 27 Feb 2021 01:06:13 +0000 (UTC) (envelope-from hackers@freebsd.org) Received: from mail0.prysmiangroup.pw (hwsrv-833329.hostwindsdns.com [104.168.145.36]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DnSyY4gvjz3R2m for ; Sat, 27 Feb 2021 01:06:13 +0000 (UTC) (envelope-from hackers@freebsd.org) From: hackers@freebsd.org To: hackers@freebsd.org Subject: New app(s) have access to your hackers@freebsd.org Date: 27 Feb 2021 01:06:12 +0000 Message-ID: <20210227010612.B6188ACFDF22CB7F@freebsd.org> X-Rspamd-Queue-Id: 4DnSyY4gvjz3R2m X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:54290, ipnet:104.168.128.0/17, country:US] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2021 01:06:13 -0000