Date: Thu, 21 Apr 2022 17:58:34 -0400 From: Rich <rincebrain@gmail.com> To: Alan Somers <asomers@freebsd.org> Cc: Michael Jung <mikej@paymentallianceintl.com>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: ZFS reservations for type=volume Message-ID: <CAOeNLuoopG9nWe9pX%2BLAHRgqBoygvczsnuE4vAfeQ4uRcgBkKw@mail.gmail.com> In-Reply-To: <CAOtMX2hvzCEFkx0CPzxc9U9Pgtpdjpk4X_zpVMc9PgC5dg03Gg@mail.gmail.com> References: <94c10113c3964b1bbd3f4ddf934c23cd@MAIL-HUB.pai.local> <CAOtMX2jyBH7k1rGZdP3PvHs7j8k=p97XOqJ9GyORaDHxavKtYQ@mail.gmail.com> <ac2c7e7344d949d49c5ca56417088d53@MAIL-HUB.pai.local> <CAOtMX2hvzCEFkx0CPzxc9U9Pgtpdjpk4X_zpVMc9PgC5dg03Gg@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] (My FBSD dev env isn't booted, but Linux and FBSD are the same in this regard...) # zpool list workspace NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED - # zfs create workspace/testme # zfs list workspace workspace/testme NAME USED AVAIL REFER MOUNTPOINT workspace 13.0T 21.5T 48.9G /workspace workspace/testme 96K 21.5T 96K /workspace/testme # df -h /workspace/testme /workspace/ Filesystem Size Used Avail Use% Mounted on workspace/testme 22T 128K 22T 1% /workspace/testme workspace 22T 49G 22T 1% /workspace # zfs set reservation=1T workspace/testme # zpool list workspace NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED - # zfs list workspace NAME USED AVAIL REFER MOUNTPOINT workspace 14.0T 20.5T 48.9G /workspace workspace/testme 96K 21.5T 96K /workspace/testme # df -h /workspace/testme /workspace/ Filesystem Size Used Avail Use% Mounted on workspace/testme 22T 128K 22T 1% /workspace/testme workspace 21T 49G 21T 1% /workspace # That is, AIUI, reservation counts against the USED/AVAIL/FREE shown by "zfs list" or "df" of everything not the reservation-holder , not against the "ALLOC" shown by zpool list...unless it's actually allocated, of course. - Rich On Thu, Apr 21, 2022 at 5:46 PM Alan Somers <asomers@freebsd.org> wrote: > Isn't 1.52T the "AVAIL" value, and 24.6T is the "USED"? > > On Thu, Apr 21, 2022 at 2:56 PM Michael Jung > <mikej@paymentallianceintl.com> wrote: > > > > Ok that makes sense... so even though running ZFS since 9.x I'll ask the > newbie question.. > > > > If it contributes to the parents "used" value why is not reflected here > in "USED" or in "AVAIL"? > > > > I would expect USED to be my reservation amount of 15.6T + whatever > space was > > being used by other things on the pool. "USED" still sits at 1.52T. > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE > > > > Thanks again. > > > > -----Original Message----- > > From: Alan Somers [mailto:asomers@freebsd.org] > > Sent: Thursday, April 21, 2022 4:40 PM > > To: Michael Jung <mikej@paymentallianceintl.com> > > Cc: freebsd-fs <freebsd-fs@freebsd.org> > > Subject: Re: ZFS reservations for type=volume > > > > A dataset's reservation is local. It doesn't contribute to its parent's > reservation. Otherwise, you wouldn't be able to separately set a > reservation on the parent. But it _does_ contribute to the parent's "used" > value. In that way, you're prevented from reserving too much data on the > parent's children. > > -Alan > > > > On Thu, Apr 21, 2022 at 2:28 PM Michael Jung < > mikej@paymentallianceintl.com> wrote: > > > > > > I have a zfs block dataset raid-5400-1/esxi-store1 that I share as an > > > iscsi target and that works great. I have set a reservation > > > > > > on that block device equal to its size so that is not sparse and thus > > > while I could over provision guests on the provided LUN, > > > > > > the storage presented as the LUN capacity will always be available. At > least this is what I want to achieve. > > > > > > > > > > > > What I find strange is that the reservation does not seem to be > > > applied to the ZFS pool ‘raid-5400-1’. Do you really need > > > > > > to set your maximum reservation at the pool level, and then apply > > > reservations to all datasets on that volume? And if so > > > > > > I would assume you could never set reservations for datasets totaling > more than what was reserved for the pool ‘raid-5400-1’. > > > > > > > > > > > > I could build out a test environment and figure out constraints but > I’d really like to know the “how it is supposed to work” > > > > > > not the “how I find it to work”. > > > > > > > > > > > > Thanks in advance. > > > > > > > > > > > > FreeBSD 14.0-CURRENT #4 main-n253875-8e72f458c6d: > > > > > > > > > > > > (this is a raidz2 pool – not my draid pool) > > > > > > > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zfs list -o > > > name,type,used,avail,ratio,compression,reservation,refreservation,vols > > > ize raid-5400-1 > > > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > > > > > raid-5400-1 filesystem 18.0T 2.08T 1.36x on none none - <- no > reservation @pool > > > > > > root@draid:/usr/src/contrib/bearssl # > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zfs list -o > > > name,type,used,avail,ratio,compression,reservation,refreservation,vols > > > ize raid-5400-1/esxi-store1 > > > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > > > > > raid-5400-1/esxi-store1 volume 16.9T 18.5T 1.78x zstd 15.6T 16.9T > 15.6T <- reservation @dataset > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zpool list > > > > > > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > > > > > > ccache 9.50G 9.10G 406M - - 88% 95% 1.00x ONLINE - > > > > > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE - <- Free does > not reflect reservation @pool > > > > > > tank 18.5T 605G 17.9T - - 0% 3% 1.00x ONLINE - > > > > > > zfsroot 103G 33.3G 69.7G - - 31% 32% 1.00x ONLINE - > > > > > > root@draid:/usr/src/contrib/bearssl # > > > > > > > > > > > > > > > > > > CONFIDENTIALITY NOTE: This message is intended only for the use of the > > > individual or entity to whom it is addressed and may contain > > > information that is privileged, confidential, and exempt from > > > disclosure under applicable law. If the reader of this message is not > > > the intended recipient, you are hereby notified that any > > > dissemination, distribution or copying of this communication is > > > strictly prohibited. If you have received this transmission in error, > > > please notify us by telephone at (502) 212-4000 or notify us at: PAI, > > > Dept. 99, > > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245 > > > > > > > > > > > > > > > > > > Disclaimer > > > > > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and may > be unlawful. > > > > > > This email has been scanned for viruses and malware, and may have been > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. > > > > > > > > > > CONFIDENTIALITY NOTE: This message is intended only for the use > > of the individual or entity to whom it is addressed and may > > contain information that is privileged, confidential, and > > exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby > > notified that any dissemination, distribution or copying > > of this communication is strictly prohibited. If you have > > received this transmission in error, please notify us by > > telephone at (502) 212-4000 or notify us at PAI, Dept. 99, > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245 > > > > > > Disclaimer > > > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and may > be unlawful. > > > > This email has been scanned for viruses and malware, and may have been > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. > > [-- Attachment #2 --] <div dir="ltr"><div>(My FBSD dev env isn't booted, but Linux and FBSD are the same in this regard...)</div><div><br></div># zpool list workspace<br>NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br>workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED -<br># zfs create workspace/testme<br># zfs list workspace workspace/testme<br>NAME USED AVAIL REFER MOUNTPOINT<br>workspace 13.0T 21.5T 48.9G /workspace<div>workspace/testme 96K 21.5T 96K /workspace/testme<br># df -h /workspace/testme /workspace/<br>Filesystem Size Used Avail Use% Mounted on<br>workspace/testme 22T 128K 22T 1% /workspace/testme<br>workspace 22T 49G 22T 1% /workspace<br><div># zfs set reservation=1T workspace/testme<br># zpool list workspace<br>NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br>workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED -<br><div># zfs list workspace<br>NAME USED AVAIL REFER MOUNTPOINT<br>workspace 14.0T 20.5T 48.9G /workspace</div><div>workspace/testme 96K 21.5T 96K /workspace/testme<br># df -h /workspace/testme /workspace/<br>Filesystem Size Used Avail Use% Mounted on<br>workspace/testme 22T 128K 22T 1% /workspace/testme<br>workspace 21T 49G 21T 1% /workspace<br>#</div><div><br></div><div>That is, AIUI, reservation counts against the USED/AVAIL/FREE shown by "zfs list" or "df" of everything not the reservation-holder , not against the "ALLOC" shown by zpool list...unless it's actually allocated, of course.</div><div><br></div><div>- Rich</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 21, 2022 at 5:46 PM Alan Somers <<a href="mailto:asomers@freebsd.org">asomers@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Isn't 1.52T the "AVAIL" value, and 24.6T is the "USED"?<br> <br> On Thu, Apr 21, 2022 at 2:56 PM Michael Jung<br> <<a href="mailto:mikej@paymentallianceintl.com" target="_blank">mikej@paymentallianceintl.com</a>> wrote:<br> ><br> > Ok that makes sense... so even though running ZFS since 9.x I'll ask the newbie question..<br> ><br> > If it contributes to the parents "used" value why is not reflected here in "USED" or in "AVAIL"?<br> ><br> > I would expect USED to be my reservation amount of 15.6T + whatever space was<br> > being used by other things on the pool. "USED" still sits at 1.52T.<br> ><br> > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE<br> > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE<br> ><br> > Thanks again.<br> ><br> > -----Original Message-----<br> > From: Alan Somers [mailto:<a href="mailto:asomers@freebsd.org" target="_blank">asomers@freebsd.org</a>]<br> > Sent: Thursday, April 21, 2022 4:40 PM<br> > To: Michael Jung <<a href="mailto:mikej@paymentallianceintl.com" target="_blank">mikej@paymentallianceintl.com</a>><br> > Cc: freebsd-fs <<a href="mailto:freebsd-fs@freebsd.org" target="_blank">freebsd-fs@freebsd.org</a>><br> > Subject: Re: ZFS reservations for type=volume<br> ><br> > A dataset's reservation is local. It doesn't contribute to its parent's reservation. Otherwise, you wouldn't be able to separately set a reservation on the parent. But it _does_ contribute to the parent's "used" value. In that way, you're prevented from reserving too much data on the parent's children.<br> > -Alan<br> ><br> > On Thu, Apr 21, 2022 at 2:28 PM Michael Jung <<a href="mailto:mikej@paymentallianceintl.com" target="_blank">mikej@paymentallianceintl.com</a>> wrote:<br> > ><br> > > I have a zfs block dataset raid-5400-1/esxi-store1 that I share as an<br> > > iscsi target and that works great. I have set a reservation<br> > ><br> > > on that block device equal to its size so that is not sparse and thus<br> > > while I could over provision guests on the provided LUN,<br> > ><br> > > the storage presented as the LUN capacity will always be available. At least this is what I want to achieve.<br> > ><br> > ><br> > ><br> > > What I find strange is that the reservation does not seem to be<br> > > applied to the ZFS pool ‘raid-5400-1’. Do you really need<br> > ><br> > > to set your maximum reservation at the pool level, and then apply<br> > > reservations to all datasets on that volume? And if so<br> > ><br> > > I would assume you could never set reservations for datasets totaling more than what was reserved for the pool ‘raid-5400-1’.<br> > ><br> > ><br> > ><br> > > I could build out a test environment and figure out constraints but I’d really like to know the “how it is supposed to work”<br> > ><br> > > not the “how I find it to work”.<br> > ><br> > ><br> > ><br> > > Thanks in advance.<br> > ><br> > ><br> > ><br> > > FreeBSD 14.0-CURRENT #4 main-n253875-8e72f458c6d:<br> > ><br> > ><br> > ><br> > > (this is a raidz2 pool – not my draid pool)<br> > ><br> > ><br> > ><br> > ><br> > ><br> > > root@draid:/usr/src/contrib/bearssl # zfs list -o<br> > > name,type,used,avail,ratio,compression,reservation,refreservation,vols<br> > > ize raid-5400-1<br> > ><br> > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE<br> > ><br> > > raid-5400-1 filesystem 18.0T 2.08T 1.36x on none none - <- no reservation @pool<br> > ><br> > > root@draid:/usr/src/contrib/bearssl #<br> > ><br> > ><br> > ><br> > > root@draid:/usr/src/contrib/bearssl # zfs list -o<br> > > name,type,used,avail,ratio,compression,reservation,refreservation,vols<br> > > ize raid-5400-1/esxi-store1<br> > ><br> > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE<br> > ><br> > > raid-5400-1/esxi-store1 volume 16.9T 18.5T 1.78x zstd 15.6T 16.9T 15.6T <- reservation @dataset<br> > ><br> > ><br> > ><br> > > root@draid:/usr/src/contrib/bearssl # zpool list<br> > ><br> > > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br> > ><br> > > ccache 9.50G 9.10G 406M - - 88% 95% 1.00x ONLINE -<br> > ><br> > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE - <- Free does not reflect reservation @pool<br> > ><br> > > tank 18.5T 605G 17.9T - - 0% 3% 1.00x ONLINE -<br> > ><br> > > zfsroot 103G 33.3G 69.7G - - 31% 32% 1.00x ONLINE -<br> > ><br> > > root@draid:/usr/src/contrib/bearssl #<br> > ><br> > ><br> > ><br> > ><br> > ><br> > > CONFIDENTIALITY NOTE: This message is intended only for the use of the<br> > > individual or entity to whom it is addressed and may contain<br> > > information that is privileged, confidential, and exempt from<br> > > disclosure under applicable law. If the reader of this message is not<br> > > the intended recipient, you are hereby notified that any<br> > > dissemination, distribution or copying of this communication is<br> > > strictly prohibited. If you have received this transmission in error,<br> > > please notify us by telephone at (502) 212-4000 or notify us at: PAI,<br> > > Dept. 99,<br> > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245<br> > ><br> > ><br> > ><br> > ><br> > ><br> > > Disclaimer<br> > ><br> > > The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.<br> > ><br> > > This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.<br> ><br> ><br> ><br> ><br> > CONFIDENTIALITY NOTE: This message is intended only for the use<br> > of the individual or entity to whom it is addressed and may<br> > contain information that is privileged, confidential, and<br> > exempt from disclosure under applicable law. If the reader<br> > of this message is not the intended recipient, you are hereby<br> > notified that any dissemination, distribution or copying<br> > of this communication is strictly prohibited. If you have<br> > received this transmission in error, please notify us by<br> > telephone at (502) 212-4000 or notify us at PAI, Dept. 99,<br> > 2101 High Wickham Place, Suite 101, Louisville, KY 40245<br> ><br> ><br> > Disclaimer<br> ><br> > The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.<br> ><br> > This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.<br> <br> </blockquote></div>help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOeNLuoopG9nWe9pX%2BLAHRgqBoygvczsnuE4vAfeQ4uRcgBkKw>