From nobody Tue May 14 01:54:08 2024 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VdfYD1hDQz5K6vH for ; Tue, 14 May 2024 01:54:24 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VdfYC3gGQz4NPd; Tue, 14 May 2024 01:54:23 +0000 (UTC) (envelope-from rlibby@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of rlibby@gmail.com designates 209.85.208.174 as permitted sender) smtp.mailfrom=rlibby@gmail.com Received: by mail-lj1-f174.google.com with SMTP id 38308e7fff4ca-2e6792ea67dso43729461fa.3; Mon, 13 May 2024 18:54:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715651661; x=1716256461; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c6ZU9TYckcH5e/LWVtAxPbydorDUoeiVpQ1VHmAJfUg=; b=n6mo5jCecSpVYjH9mUXJpoVJlAWKmwi7+JXLDurAUhHunSM2HoJPmha4/Rw7WyrNr9 ++hnAQUO9vHTrRMUkkxX1AHHhHQ5+f918Rqd569SUsIKhsU1YPWNZcFA4D/J3sZT6izv zTOcx7aA32zYj6/wHKKHYbA83GUeoBZGXG8NDt1tP1LG+q0LWGRENMZU0WDaOi7qiOKb nlOeagvmprmrPNY8nyYWtPcxvkVNaNqrHfa7J1+rTTJGJEBGZdj4D6iE6RY4d2LbG0VS qyhANMrh2bRx6weAncxyKbl35Yl7ZJnD991srM1l5f57cXIqN+HRPcFq6osEMQC530Zk 7tjA== X-Forwarded-Encrypted: i=1; AJvYcCWM7mraU0fIj2UhjLOg+73lriCY7WzUyH6IPtmmYKkiZBA+Jr2dEXd1WYirbFQZfSOjlH40eu2bDVpjbrRnxgnMyDsZ5NuOSkMf3xvtfdO78Wf6+Ts= X-Gm-Message-State: AOJu0Yyh4BQ6wsYeMtah2Fo3/l9yVPTJmf5z5idKKYgn2qwZ0xdKNYG3 cKFHRLzxNaSddzHaOPYYPsust+rPoyG1bWYuUR9J9JNhLQ03fXCWqfBs5r+J X-Google-Smtp-Source: AGHT+IHivBTRnX0mOzEyLOQAmtEqd49lWGvyuTweIXwMac5nht5crIdDMIOAHDqj65oYKoqh0Gowfw== X-Received: by 2002:a2e:7a0e:0:b0:2e2:466b:1a56 with SMTP id 38308e7fff4ca-2e52039e2c3mr84236131fa.53.1715651660451; Mon, 13 May 2024 18:54:20 -0700 (PDT) Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com. [209.85.208.175]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2e4d1515664sm16332461fa.93.2024.05.13.18.54.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 13 May 2024 18:54:19 -0700 (PDT) Received: by mail-lj1-f175.google.com with SMTP id 38308e7fff4ca-2e576057c2bso40375961fa.1; Mon, 13 May 2024 18:54:19 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCUdq88TCTk+tFmXHgC8IZMYPRZZt6eDw/1xa7JYY8Pk82hvy7VJFTXpn3hmMBe0MwE5ksDlbXtCvKLDihm7b3C4irYbSg92ptD8lZHPRg3wy+4GF7Y= X-Received: by 2002:a2e:8014:0:b0:2e4:a8c3:7618 with SMTP id 38308e7fff4ca-2e51fd45319mr86891961fa.16.1715651659672; Mon, 13 May 2024 18:54:19 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <0a3ddc685e54a289ff5cff569a95cd29@Leidinger.net> <5c7357c3-5a10-4a8e-9245-8a5787c57f35@freebsd.org> In-Reply-To: From: Ryan Libby Date: Mon, 13 May 2024 18:54:08 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Graph of the FreeBSD memory fragmentation To: Alexander Leidinger Cc: =?UTF-8?Q?Bojan_Novkovi=C4=87?= , Current , alc@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.93)[-0.929]; FORGED_SENDER(0.30)[rlibby@freebsd.org,rlibby@gmail.com]; NEURAL_HAM_SHORT(-0.27)[-0.270]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEFALL_USER(0.00)[rlibby]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.174:from,209.85.208.175:received]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.174:from]; MISSING_XM_UA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rlibby@freebsd.org,rlibby@gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; R_DKIM_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[4] X-Rspamd-Queue-Id: 4VdfYC3gGQz4NPd On Thu, May 9, 2024 at 2:36=E2=80=AFAM Alexander Leidinger wrote: > > Am 2024-05-08 18:45, schrieb Bojan Novkovi=C4=87: > > Hi, > > > > On 5/7/24 14:02, Alexander Leidinger wrote: > > > >> Hi, > >> > >> I created some graphs of the memory fragmentation. > >> https://www.leidinger.net/blog/2024/05/07/plotting-the-freebsd-memory-= fragmentation/ > >> > >> My goal was not comparing a specific change on a given benchmark, but > >> to "have something which visualizes memory fragmentation". As part of > >> that, Bojans commit > >> https://cgit.freebsd.org/src/commit/?id=3D7a79d066976149349ecb90240d02= eed0c4268737 > >> was just in the middle of my data collection. I have the impression > >> that it made a positive difference in my non deterministic workload. > > Thank you for working on this, the plots look great! > > They provide a really clean visual overview of what's happening. > > I'm working on another type of memory visualization which might > > interest you, I'll share it with you once its done. > > One small nit - the fragmentation index does not quantify fragmentation > > for UMA buckets, but for page allocator freelists. > > Do I get it more correctly now: UMA buckets are type/structure specific > allocation lists, and the page allocator freelists are size-specific > allocation lists (which are used by UMA when no free item is available > in a bucket)? > Yeah, that's more correct. UMA is a higher level allocator than the vm phys free lists. The latter is what the proposed sysctl measures. That measurement is not directly related to UMA or how UMA allocates pages for most zones, except the handful of UMA_ZONE_CONTIG zones. Because otherwise, UMA doesn't explicitly do or require allocations of contiguous pages. Most allocations by UMA of backing pages are done as single pages from the perspective of the vm phys free lists. When possible (slab size of one page and machine architecture support) it then uses direct map addresses to refer to items. Otherwise the backing pages (single or multiple not-necessarily-contiguous) are mapped into kernel virtual address space. Re malloc(9), "small" (up to 64 KB) allocations are served from uma zone "buckets" of various sizes, while larger allocations skip UMA and allocate kmem directly in a similar way to how UMA does allocations for large slabs (as in multiple individual pages are mapped). UMA "buckets" in the sense of struct uma_bucket are caches of free items. For non-cache zones, if buckets are exhausted then UMA goes to slabs. If slabs are exhausted then UMA allocates pages. Pages will first be served from the page cache, before the vm phys free lists. That was a long winded way of saying: the "UMA bucket" axis is actually "vm phys free list order". That said, I find that dimension confusing because in fact there's just one piece of information there, the average size of a free list entry, and it doesn't actually depend on the free list order. The graph could be 2D. The paper that defines this fragmentation index also says that "the fragmentation index is only meaningful when an allocation fails". Are you actually seeing any contiguous allocations failures in your measurements? Without that context, it seems like what the proposed sysctl reports is indirectly just the average size of free list entries. We could just report that. > >> Is there anything which prevents https://reviews.freebsd.org/D40575 to > >> be committed? > > D40575 is closely tied to the compaction patch (D40772) which is > > currently on hold until another issue is solved (see D45046 and related > > revisions for more details). > > Any idea about https://reviews.freebsd.org/D16620 ? Is D45046 supposed > to replace this, or is it about something else? > I wanted to try D16620, but it doesn't apply and my naive/mechanical way > of applying it panics. > > > I didn't consider landing D40575 because of that, but I guess it could > > be useful on its own. > > It at least gives a way to quantify with numbers resp. qualitatively > visualize. And as such it may help in visualizing differences like with > your guard-pages commit. I wonder if the segregation of nofree > allocations may result in a similar improvement for long-running > systems. > > Bye, > Alexander. > > -- > http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF Ryan