From nobody Thu Feb 2 16:17:09 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P73pV3SGXz3fpVx for ; Thu, 2 Feb 2023 16:17:22 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P73pV1MZhz3jMt for ; Thu, 2 Feb 2023 16:17:22 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-io1-xd29.google.com with SMTP id r6so928999ioj.5 for ; Thu, 02 Feb 2023 08:17:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20210112.gappssmtp.com; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=/1P19ak9VIN28ud035JOAJZLnXcvlEYRFxrsJB1xNxc=; b=v4CgXKG5TaVg4WKWVEjWJlXeEP3eHNsEsQtw+d7TrY1ZzP5oGwsDuT+CbAzpG1JrRf Cpif/wLciuEywljvTGZjD10NTufrCGL5VqjMbx/6w6e61TzXhTPcVDfQpkdcD6o91jfI Q7fPg8uXt9a8zkAg04hB7eq+QNXNLYneOamDqoAcSn2wXq7Avcq601zwrnBPatYpD5Gq gv6Iefh3ELxSL4T5Vgzb4TtTZtBUzuNRqTYQyQ/xNXNeo3PhaKulRFLA+aACpedM6b9D V+LwoOXCfJVB0avBUalvRo5tAtLb+wXzZR++8ubsu5fQ3A15Gw8QcnlIPA+Qny8aY7ug ikdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/1P19ak9VIN28ud035JOAJZLnXcvlEYRFxrsJB1xNxc=; b=4cZNvlgM3XODSa/DfGZibGqoJzgIV/v/+gXinKE3WeE+Oa6MBWqWbH4DdRP/epWjFZ LsRQ43jjEqTcMa/0SdQN/Q2gGXEZLoxoL1w9SeDhL5zgwpVOdk2ACGjz5FRmGymN9ACG 9TaCkHdic+tPoZS2kGKyiN1HW6J/RsjkU0kTXUv1E1K6EfvqRXK1Q+nYn4uIxJCDZMDY ounltGqQiHXmdlTmd2k/0WkdY8AOKLhB22LsOn6+cdlEHtFw2fGhBPtdtnk/c2aKY9Ss whA/RHfTz17NSiS+jmxjaP5HTgG4TLB/JhQkTWn1tEZqZa2S49EWwW4yOuDaFhAKhKXJ vlzQ== X-Gm-Message-State: AO0yUKWBv7Gz/NFkMzh7KurCw3RHuHCWonLb3R4eDdnxmLSSf64Ci1Sm TexcplCnUqZaJ7SHu5Ji31yrAA5N2Vm+1PoE X-Google-Smtp-Source: AK7set/wyGkKON4BA8lOTqvCq0K8Wv8gXTiMb5DrEA15qoINP5JOHrTpAOLcVQObYkkJCnDTcfYeWQ== X-Received: by 2002:a05:6602:2116:b0:6de:3e2c:d791 with SMTP id x22-20020a056602211600b006de3e2cd791mr4748797iox.1.1675354641140; Thu, 02 Feb 2023 08:17:21 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id h26-20020a02c73a000000b003b778515852sm17416jao.168.2023.02.02.08.17.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 02 Feb 2023 08:17:20 -0800 (PST) From: Bakul Shah Message-Id: <3DEFA166-79E8-49AD-B0DD-7E524C8EAAE8@iitbombay.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Swap, ZFS & ARC Date: Thu, 2 Feb 2023 08:17:09 -0800 In-Reply-To: Cc: "hackers@freebsd.org" To: jbo@insane.engineer References: X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4P73pV1MZhz3jMt X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Feb 2, 2023, at 6:28 AM, jbo@insane.engineer wrote: >=20 > Hello folks, >=20 > Based on a discussion on the forums not so long ago I tried to figure = out how swap usage on a ZFS system plays together with ARC. However, I = could find very little to no information on this which leads me to = believe that there is some "core concept" I might be oblivious to. >=20 > The main question is basically this: Your system starts to swap out = data from RAM to your swap partition. This swap data on disk ultimately = resides somewhere in a ZFS pool. If this data then gets accessed, it = might be cached by ARC essentially eating up memory again which seems = counter productive. > Is there any magic which prevents swap partitions from being loaded = into ARC? Or is this a non-issue for some other reason? I suspect this bug affects FreeBSD as well: https://github.com/openzfs/zfs/issues/7734 =46rom https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279 I'm not an expert in this area of the code, but I think that swap on = ZVOL is inherently unreliable due to writes to the swap ZVOL having to = go through the normal TXG sync and ZIO write paths, which can require = lots of memory allocations by design (and these memory allocations can = stall due to a low memory situation). I believe this to be true for swap = on ZVOL for illumos, as well as Linux, and presumably FreeBSD too = (although I have no experience using it on FreeBSD, so I could be = wrong). FYI. I don't know enough about zfs internals so can't say if this poster = is right or not but I too have just used a disk partition as opposed to = a zvolume for swap.= --Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Feb 2, 2023, at 6:28 AM, = jbo@insane.engineer wrote:

Hello folks,

Based on a = discussion on the forums not so long ago I tried to figure out how swap = usage on a ZFS system plays together with ARC. However, I could find = very little to no information on this which leads me to believe that = there is some "core concept" I might be oblivious to.

The main = question is basically this: Your system starts to swap out data from RAM = to your swap partition. This swap data on disk ultimately resides = somewhere in a ZFS pool. If this data then gets accessed, it might be = cached by ARC essentially eating up memory again which seems counter = productive.
Is there any magic which prevents swap partitions from = being loaded into ARC? Or is this a non-issue for some other = reason?

I suspect this bug affects FreeBSD as = well:

https://github.com/openzfs/zfs/issues/7734

From = https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279

<= blockquote style=3D"margin: 0 0 0 40px; border: none; padding: = 0px;">
I'm not an expert in this area of the code, but I think that = swap on ZVOL is inherently unreliable due to writes to the swap ZVOL = having to go through the normal TXG sync and ZIO write paths, which can = require lots of memory allocations by design (and these memory = allocations can stall due to a low memory situation). I believe this to = be true for swap on ZVOL for illumos, as well as Linux, and presumably = FreeBSD too (although I have no experience using it on FreeBSD, so I = could be wrong).

FYI. I don't know enough = about zfs internals so can't say if this poster is right or not but I = too have just used a disk partition as opposed to a zvolume for = swap.
= --Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9--