Date: Thu, 2 Feb 2023 09:52:34 -0700 From: Warner Losh <imp@bsdimp.com> To: Bakul Shah <bakul@iitbombay.org> Cc: jbo@insane.engineer, "hackers@freebsd.org" <hackers@freebsd.org> Subject: Re: Swap, ZFS & ARC Message-ID: <CANCZdfraCA5O7OonRbjboNoz54Dwqwxx6h0Qbj-UHjfhDvG9PA@mail.gmail.com> In-Reply-To: <3DEFA166-79E8-49AD-B0DD-7E524C8EAAE8@iitbombay.org> References: <ysSGWaXF7vCDubzgFdov_9k6SMLNdZ3-zHV-2LOraYOxV_8yoAkDVPqDSicYsgBw1KdL6BjP4FkK1KFs7NKZ4RRXcSDxVe_Qy_i7b_5AbSI=@insane.engineer> <3DEFA166-79E8-49AD-B0DD-7E524C8EAAE8@iitbombay.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000dab04b05f3ba6460 Content-Type: text/plain; charset="UTF-8" On Thu, Feb 2, 2023 at 9:17 AM Bakul Shah <bakul@iitbombay.org> wrote: > > On Feb 2, 2023, at 6:28 AM, jbo@insane.engineer wrote: > > > Hello folks, > > Based on a discussion on the forums not so long ago I tried to figure out > how swap usage on a ZFS system plays together with ARC. However, I could > find very little to no information on this which leads me to believe that > there is some "core concept" I might be oblivious to. > > The main question is basically this: Your system starts to swap out data > from RAM to your swap partition. This swap data on disk ultimately resides > somewhere in a ZFS pool. If this data then gets accessed, it might be > cached by ARC essentially eating up memory again which seems counter > productive. > Is there any magic which prevents swap partitions from being loaded into > ARC? Or is this a non-issue for some other reason? > > > I suspect this bug affects FreeBSD as well: > > https://github.com/openzfs/zfs/issues/7734 > > From https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279 > > I'm not an expert in this area of the code, but I think that swap on ZVOL > is inherently unreliable due to writes to the swap ZVOL having to go > through the normal TXG sync and ZIO write paths, which can require lots of > memory allocations by design (and these memory allocations can stall due to > a low memory situation). I believe this to be true for swap on ZVOL for > illumos, as well as Linux, and presumably FreeBSD too (although I have no > experience using it on FreeBSD, so I could be wrong). > > > FYI. I don't know enough about zfs internals so can't say if this poster > is right or not but I too have just used a disk partition as opposed to a > zvolume for swap. > They have it right. FreeBSD tries hard (though isn't perfect in this regard) to not allocate memory in the I/O path, especially if it knows the request is for swap. It's possible to swap to zvols, and it actually works when the memory pressures are light (since the stalls usually aren't too long and usually complete). When memory pressure is high, though, these can turn into deadlocks. Warner --000000000000dab04b05f3ba6460 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Thu, Feb 2, 2023 at 9:17 AM Bakul = Shah <<a href=3D"mailto:bakul@iitbombay.org">bakul@iitbombay.org</a>>= wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><di= v><br></div>On Feb 2, 2023, at 6:28 AM, jbo@insane.engineer wrote:<br><bloc= kquote type=3D"cite"><br>Hello folks,<br><br>Based on a discussion on the f= orums not so long ago I tried to figure out how swap usage on a ZFS system = plays together with ARC. However, I could find very little to no informatio= n on this which leads me to believe that there is some "core concept&q= uot; I might be oblivious to.<br><br>The main question is basically this: Y= our system starts to swap out data from RAM to your swap partition. This sw= ap data on disk ultimately resides somewhere in a ZFS pool. If this data th= en gets accessed, it might be cached by ARC essentially eating up memory ag= ain which seems counter productive.<br>Is there any magic which prevents sw= ap partitions from being loaded into ARC? Or is this a non-issue for some o= ther reason?<br></blockquote><br>I suspect this bug affects FreeBSD as well= :<div><br><a href=3D"https://github.com/openzfs/zfs/issues/7734" target=3D"= _blank">https://github.com/openzfs/zfs/issues/7734</a><br><br>From=C2=A0<a = href=3D"https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279" = target=3D"_blank">https://github.com/openzfs/zfs/issues/7734#issuecomment-4= 22082279</a><br><br><blockquote style=3D"margin:0px 0px 0px 40px;border:non= e;padding:0px"><div>I'm not an expert in this area of the code, but I t= hink that swap on ZVOL is inherently unreliable due to writes to the swap Z= VOL having to go through the normal TXG sync and ZIO write paths, which can= require lots of memory allocations by design (and these memory allocations= can stall due to a low memory situation). I believe this to be true for sw= ap on ZVOL for illumos, as well as Linux, and presumably FreeBSD too (altho= ugh I have no experience using it on FreeBSD, so I could be wrong).</div></= blockquote><br><div>FYI. I don't know enough about zfs internals so can= 't say if this poster is right or not but I too have just used a disk p= artition as opposed to a zvolume for swap.</div></div></div></blockquote><d= iv><br></div><div>They have it right. FreeBSD tries hard (though isn't = perfect in this regard) to not allocate memory in the I/O path, especially = if it knows the request is for swap. It's possible to swap to zvols, an= d it actually works when the memory pressures are light (since the stalls u= sually aren't too long and usually complete). When memory pressure is h= igh, though, these can turn into deadlocks.</div><div><br></div><div>Warner= </div></div></div> --000000000000dab04b05f3ba6460--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfraCA5O7OonRbjboNoz54Dwqwxx6h0Qbj-UHjfhDvG9PA>