From owner-freebsd-current@freebsd.org Sun Jun 24 20:07:14 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6EAB100C7CB for ; Sun, 24 Jun 2018 20:07:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-4.consmr.mail.bf2.yahoo.com (sonic302-4.consmr.mail.bf2.yahoo.com [74.6.135.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A93475103 for ; Sun, 24 Jun 2018 20:07:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: G7mIGs0VM1lmUMpe_iB8gW4DbxHAPRsNd8uUEJOUUVSbPOAVwRffDFw3TnggecL 0ZypAZOgZo6FzbVpzNRKzZPYAn1IWwGuxDIEdo9_tgKIMdMzyVLZ1.SWdIDjRodk9uOXxy136EtH IN2El16qyAttF0RALEy2Q2LaNZjf28L2E6JjtjpSmDZxQzpqE1TkWqIbincYvMeeLgYIai.D5lgL tS0rE3g1eCKE6HcI0gLXGA2fYtAz21_U3snTvlZ.vt7Svv_1kqScaOtadI5nsqJZEf4J7xEz2ZeA L9KtYz2v.e8g9Ln3uedfrE7T.l_qApgXgrTzc8Dv4iEk.jxcCSVI8rXrj2tzHz6OZt9X6pb0hG18 KjQkw1QWpybu90PG5RQaf.61UKWfV6DYKbKkkqDsTUVd8c3Oz7hqbiCw_wOF3JnZkTM_PU4sb03c rNQyndiq9R8zBoceOWBTOL69O7y1j_GeMu4EYAzUzU1MR33KfErlFquCs.G.NqXRlGmtEcQXRjvR soIMCJmjz9ch4ScQ1rzskG_kHEpmhDhuToPlnN0U0U20Svq7kLGtZUZ1mGzo52vaD5zX6UKglRML fVN6lNJE1LsheBDF4BgoPitvkEnaAdyY641dF1IPV3McL4cmFKh3Pmwkpv0Krj_.PquR6qrO0ibx b8vnbpXg0ksEf3RuNuTwDOutseltFHuO3WOwevqcDC.HLys_XMXQFD3g- Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.bf2.yahoo.com with HTTP; Sun, 24 Jun 2018 20:07:06 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp430.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 318736ef42c27f6b42c945aefdb6cb32; Sun, 24 Jun 2018 20:07:03 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 11.4 \(3445.8.2\)) Subject: Re: numa involved in instability and swap usage despite RAM free? Message-Id: <14F2E7C6-4094-4EF5-A1D7-0D56B43ACE62@yahoo.com> Date: Sun, 24 Jun 2018 13:07:00 -0700 To: Alexander@leidinger.net, FreeBSD Current X-Mailer: Apple Mail (2.3445.8.2) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jun 2018 20:07:14 -0000 Alexander Leidinger Alexander at leidinger.net wrote on Sun Jun 24 10:03:49 UTC 2018 : > Short: > Can it be that enabling numa in the kernel is the reason why some > people see instability with zfs and usage of swap while a lot of free > RAM is available? [It will likely be a few months before I again have access to the environment these notes are based on. It has been about a month since I last had access.] On a AMD Ryzen Threadripper 1950X (16 cores, 2 HW threads per core) I enabled: options NUMA options MAXMEMDOM=2 in fairly recent times. This is a UFS context, not a ZFS one. I'd not been explicitly controlling how things run (so using defaults). This is head with debugging disabled (via including GENERIC and overriding). I did not have the swap usage problem with doing many buildworld buildkernel (self hosted and the cross builds for several targets). Nor when did a poudriere bulk -a (with ALLOW_MAKE_JOBS=yes ). This was a FreeBSD native boot context at the time. (I usually have run the same drives under Hyper-V but have not seen the problem there either.) For native FreeBSD I used -j32 (in buildworld/buildkernel terms but also for the bulk -a) and for under-Hyper-V I used -j28 . 96 GiBytes of ECC RAM total (48 GiBytes/NUMA-node). I'm not sure how common NUMA being enabled is, nor how common various MAXMEMDOM settings are. I'd not be surprised if various folks reporting problems had not explicitly enabled NUMA, nor set some explicit MAXMEMDOM figure. It may be that they all have ZFS in common in fairly recent times. (I'm ignoring examples of long-latency I/O on the same device as some swap partitions that are in use: this gets into Out Of Memory process killing without the swap being mostly used. Some reports of swap problems have this sort of issue involved on small systems unlikely to be using ZFS.) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)