Date: Thu, 21 Apr 2022 17:58:34 -0400 From: Rich <rincebrain@gmail.com> To: Alan Somers <asomers@freebsd.org> Cc: Michael Jung <mikej@paymentallianceintl.com>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: ZFS reservations for type=volume Message-ID: <CAOeNLuoopG9nWe9pX%2BLAHRgqBoygvczsnuE4vAfeQ4uRcgBkKw@mail.gmail.com> In-Reply-To: <CAOtMX2hvzCEFkx0CPzxc9U9Pgtpdjpk4X_zpVMc9PgC5dg03Gg@mail.gmail.com> References: <94c10113c3964b1bbd3f4ddf934c23cd@MAIL-HUB.pai.local> <CAOtMX2jyBH7k1rGZdP3PvHs7j8k=p97XOqJ9GyORaDHxavKtYQ@mail.gmail.com> <ac2c7e7344d949d49c5ca56417088d53@MAIL-HUB.pai.local> <CAOtMX2hvzCEFkx0CPzxc9U9Pgtpdjpk4X_zpVMc9PgC5dg03Gg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000081810c05dd313637 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable (My FBSD dev env isn't booted, but Linux and FBSD are the same in this regard...) # zpool list workspace NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED - # zfs create workspace/testme # zfs list workspace workspace/testme NAME USED AVAIL REFER MOUNTPOINT workspace 13.0T 21.5T 48.9G /workspace workspace/testme 96K 21.5T 96K /workspace/testme # df -h /workspace/testme /workspace/ Filesystem Size Used Avail Use% Mounted on workspace/testme 22T 128K 22T 1% /workspace/testme workspace 22T 49G 22T 1% /workspace # zfs set reservation=3D1T workspace/testme # zpool list workspace NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED - # zfs list workspace NAME USED AVAIL REFER MOUNTPOINT workspace 14.0T 20.5T 48.9G /workspace workspace/testme 96K 21.5T 96K /workspace/testme # df -h /workspace/testme /workspace/ Filesystem Size Used Avail Use% Mounted on workspace/testme 22T 128K 22T 1% /workspace/testme workspace 21T 49G 21T 1% /workspace # That is, AIUI, reservation counts against the USED/AVAIL/FREE shown by "zfs list" or "df" of everything not the reservation-holder , not against the "ALLOC" shown by zpool list...unless it's actually allocated, of course. - Rich On Thu, Apr 21, 2022 at 5:46 PM Alan Somers <asomers@freebsd.org> wrote: > Isn't 1.52T the "AVAIL" value, and 24.6T is the "USED"? > > On Thu, Apr 21, 2022 at 2:56 PM Michael Jung > <mikej@paymentallianceintl.com> wrote: > > > > Ok that makes sense... so even though running ZFS since 9.x I'll ask th= e > newbie question.. > > > > If it contributes to the parents "used" value why is not reflected here > in "USED" or in "AVAIL"? > > > > I would expect USED to be my reservation amount of 15.6T + whatever > space was > > being used by other things on the pool. "USED" still sits at 1.52T. > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE > > > > Thanks again. > > > > -----Original Message----- > > From: Alan Somers [mailto:asomers@freebsd.org] > > Sent: Thursday, April 21, 2022 4:40 PM > > To: Michael Jung <mikej@paymentallianceintl.com> > > Cc: freebsd-fs <freebsd-fs@freebsd.org> > > Subject: Re: ZFS reservations for type=3Dvolume > > > > A dataset's reservation is local. It doesn't contribute to its parent's > reservation. Otherwise, you wouldn't be able to separately set a > reservation on the parent. But it _does_ contribute to the parent's "used= " > value. In that way, you're prevented from reserving too much data on the > parent's children. > > -Alan > > > > On Thu, Apr 21, 2022 at 2:28 PM Michael Jung < > mikej@paymentallianceintl.com> wrote: > > > > > > I have a zfs block dataset raid-5400-1/esxi-store1 that I share as an > > > iscsi target and that works great. I have set a reservation > > > > > > on that block device equal to its size so that is not sparse and thus > > > while I could over provision guests on the provided LUN, > > > > > > the storage presented as the LUN capacity will always be available. A= t > least this is what I want to achieve. > > > > > > > > > > > > What I find strange is that the reservation does not seem to be > > > applied to the ZFS pool =E2=80=98raid-5400-1=E2=80=99. Do you really = need > > > > > > to set your maximum reservation at the pool level, and then apply > > > reservations to all datasets on that volume? And if so > > > > > > I would assume you could never set reservations for datasets totaling > more than what was reserved for the pool =E2=80=98raid-5400-1=E2=80=99. > > > > > > > > > > > > I could build out a test environment and figure out constraints but > I=E2=80=99d really like to know the =E2=80=9Chow it is supposed to work= =E2=80=9D > > > > > > not the =E2=80=9Chow I find it to work=E2=80=9D. > > > > > > > > > > > > Thanks in advance. > > > > > > > > > > > > FreeBSD 14.0-CURRENT #4 main-n253875-8e72f458c6d: > > > > > > > > > > > > (this is a raidz2 pool =E2=80=93 not my draid pool) > > > > > > > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zfs list -o > > > name,type,used,avail,ratio,compression,reservation,refreservation,vol= s > > > ize raid-5400-1 > > > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > > > > > raid-5400-1 filesystem 18.0T 2.08T 1.36x on none none - <- no > reservation @pool > > > > > > root@draid:/usr/src/contrib/bearssl # > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zfs list -o > > > name,type,used,avail,ratio,compression,reservation,refreservation,vol= s > > > ize raid-5400-1/esxi-store1 > > > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > > > > > raid-5400-1/esxi-store1 volume 16.9T 18.5T 1.78x zstd 15.6T 16.9T > 15.6T <- reservation @dataset > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zpool list > > > > > > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > > > > > > ccache 9.50G 9.10G 406M - - 88% 95% 1.00x ONLINE - > > > > > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE - <- Free does > not reflect reservation @pool > > > > > > tank 18.5T 605G 17.9T - - 0% 3% 1.00x ONLINE - > > > > > > zfsroot 103G 33.3G 69.7G - - 31% 32% 1.00x ONLINE - > > > > > > root@draid:/usr/src/contrib/bearssl # > > > > > > > > > > > > > > > > > > CONFIDENTIALITY NOTE: This message is intended only for the use of th= e > > > individual or entity to whom it is addressed and may contain > > > information that is privileged, confidential, and exempt from > > > disclosure under applicable law. If the reader of this message is not > > > the intended recipient, you are hereby notified that any > > > dissemination, distribution or copying of this communication is > > > strictly prohibited. If you have received this transmission in error, > > > please notify us by telephone at (502) 212-4000 or notify us at: PAI, > > > Dept. 99, > > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245 > > > > > > > > > > > > > > > > > > Disclaimer > > > > > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and m= ay > be unlawful. > > > > > > This email has been scanned for viruses and malware, and may have bee= n > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. > > > > > > > > > > CONFIDENTIALITY NOTE: This message is intended only for the use > > of the individual or entity to whom it is addressed and may > > contain information that is privileged, confidential, and > > exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby > > notified that any dissemination, distribution or copying > > of this communication is strictly prohibited. If you have > > received this transmission in error, please notify us by > > telephone at (502) 212-4000 or notify us at PAI, Dept. 99, > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245 > > > > > > Disclaimer > > > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and m= ay > be unlawful. > > > > This email has been scanned for viruses and malware, and may have been > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. > > --00000000000081810c05dd313637 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>(My FBSD dev env isn't booted, but Linux and FBSD= =C2=A0are the same in this regard...)</div><div><br></div># zpool list work= space<br>NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE =C2=A0ALLOC =C2=A0 FREE =C2= =A0CKPOINT =C2=A0EXPANDSZ =C2=A0 FRAG =C2=A0 =C2=A0CAP =C2=A0DEDUP =C2=A0 = =C2=A0HEALTH =C2=A0ALTROOT<br>workspace =C2=A035.4T =C2=A012.9T =C2=A022.5T= =C2=A0 =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 3= % =C2=A0 =C2=A036% =C2=A01.00x =C2=A0DEGRADED =C2=A0-<br># zfs create works= pace/testme<br># zfs list workspace workspace/testme<br>NAME =C2=A0 =C2=A0 = =C2=A0 =C2=A0USED =C2=A0AVAIL =C2=A0 =C2=A0 REFER =C2=A0MOUNTPOINT<br>works= pace =C2=A013.0T =C2=A021.5T =C2=A0 =C2=A0 48.9G =C2=A0/workspace<div>works= pace/testme =C2=A0 =C2=A096K =C2=A021.5T =C2=A0 =C2=A0 =C2=A0 96K =C2=A0/wo= rkspace/testme<br># df -h /workspace/testme /workspace/<br>Filesystem =C2= =A0 =C2=A0 =C2=A0 =C2=A0Size =C2=A0Used Avail Use% Mounted on<br>workspace/= testme =C2=A0 22T =C2=A0128K =C2=A0 22T =C2=A0 1% /workspace/testme<br>work= space =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A022T =C2=A0 49G =C2=A0 22T =C2=A0 1%= /workspace<br><div># zfs set reservation=3D1T workspace/testme<br># zpool = list workspace<br>NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE =C2=A0ALLOC =C2=A0 F= REE =C2=A0CKPOINT =C2=A0EXPANDSZ =C2=A0 FRAG =C2=A0 =C2=A0CAP =C2=A0DEDUP = =C2=A0 =C2=A0HEALTH =C2=A0ALTROOT<br>workspace =C2=A035.4T =C2=A012.9T =C2= =A022.5T =C2=A0 =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 = =C2=A0 3% =C2=A0 =C2=A036% =C2=A01.00x =C2=A0DEGRADED =C2=A0-<br><div># zfs= list workspace<br>NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0USED =C2=A0AVAIL =C2=A0 = =C2=A0 REFER =C2=A0MOUNTPOINT<br>workspace =C2=A014.0T =C2=A020.5T =C2=A0 = =C2=A0 48.9G =C2=A0/workspace</div><div>workspace/testme =C2=A0 =C2=A096K = =C2=A021.5T =C2=A0 =C2=A0 =C2=A0 96K =C2=A0/workspace/testme<br># df -h /wo= rkspace/testme /workspace/<br>Filesystem =C2=A0 =C2=A0 =C2=A0 =C2=A0Size = =C2=A0Used Avail Use% Mounted on<br>workspace/testme =C2=A0 22T =C2=A0128K = =C2=A0 22T =C2=A0 1% /workspace/testme<br>workspace =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A021T =C2=A0 49G =C2=A0 21T =C2=A0 1% /workspace<br>#</div><div>= <br></div><div>That is, AIUI, reservation counts against the USED/AVAIL/FRE= E shown by "zfs list" or "df" of everything not the res= ervation-holder , not against the "ALLOC" shown by zpool list...u= nless it's actually allocated, of course.</div><div><br></div><div>- Ri= ch</div></div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" c= lass=3D"gmail_attr">On Thu, Apr 21, 2022 at 5:46 PM Alan Somers <<a href= =3D"mailto:asomers@freebsd.org">asomers@freebsd.org</a>> wrote:<br></div= ><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border= -left:1px solid rgb(204,204,204);padding-left:1ex">Isn't 1.52T the &quo= t;AVAIL" value, and 24.6T is the "USED"?<br> <br> On Thu, Apr 21, 2022 at 2:56 PM Michael Jung<br> <<a href=3D"mailto:mikej@paymentallianceintl.com" target=3D"_blank">mike= j@paymentallianceintl.com</a>> wrote:<br> ><br> > Ok that makes sense... so even though running ZFS since 9.x I'll a= sk the newbie question..<br> ><br> > If it contributes to the parents "used" value why is not ref= lected here in "USED" or in "AVAIL"?<br> ><br> > I would expect USED to be my reservation amount of 15.6T + whatever sp= ace was<br> > being used by other things on the pool. "USED" still sits at= 1.52T.<br> ><br> > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE<br> > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE<br> ><br> > Thanks again.<br> ><br> > -----Original Message-----<br> > From: Alan Somers [mailto:<a href=3D"mailto:asomers@freebsd.org" targe= t=3D"_blank">asomers@freebsd.org</a>]<br> > Sent: Thursday, April 21, 2022 4:40 PM<br> > To: Michael Jung <<a href=3D"mailto:mikej@paymentallianceintl.com" = target=3D"_blank">mikej@paymentallianceintl.com</a>><br> > Cc: freebsd-fs <<a href=3D"mailto:freebsd-fs@freebsd.org" target=3D= "_blank">freebsd-fs@freebsd.org</a>><br> > Subject: Re: ZFS reservations for type=3Dvolume<br> ><br> > A dataset's reservation is local. It doesn't contribute to its= parent's reservation. Otherwise, you wouldn't be able to separatel= y set a reservation on the parent. But it _does_ contribute to the parent&#= 39;s "used" value. In that way, you're prevented from reservi= ng too much data on the parent's children.<br> > -Alan<br> ><br> > On Thu, Apr 21, 2022 at 2:28 PM Michael Jung <<a href=3D"mailto:mik= ej@paymentallianceintl.com" target=3D"_blank">mikej@paymentallianceintl.com= </a>> wrote:<br> > ><br> > > I have a zfs block dataset raid-5400-1/esxi-store1 that I share a= s an<br> > > iscsi target and that works great. I have set a reservation<br> > ><br> > > on that block device equal to its size so that is not sparse and = thus<br> > > while I could over provision guests on the provided LUN,<br> > ><br> > > the storage presented as the LUN capacity will always be availabl= e. At least this is what I want to achieve.<br> > ><br> > ><br> > ><br> > > What I find strange is that the reservation does not seem to be<b= r> > > applied to the ZFS pool =E2=80=98raid-5400-1=E2=80=99. Do you rea= lly need<br> > ><br> > > to set your maximum reservation at the pool level, and then apply= <br> > > reservations to all datasets on that volume? And if so<br> > ><br> > > I would assume you could never set reservations for datasets tota= ling more than what was reserved for the pool =E2=80=98raid-5400-1=E2=80=99= .<br> > ><br> > ><br> > ><br> > > I could build out a test environment and figure out constraints b= ut I=E2=80=99d really like to know the =E2=80=9Chow it is supposed to work= =E2=80=9D<br> > ><br> > > not the =E2=80=9Chow I find it to work=E2=80=9D.<br> > ><br> > ><br> > ><br> > > Thanks in advance.<br> > ><br> > ><br> > ><br> > > FreeBSD 14.0-CURRENT #4 main-n253875-8e72f458c6d:<br> > ><br> > ><br> > ><br> > > (this is a raidz2 pool =E2=80=93 not my draid pool)<br> > ><br> > ><br> > ><br> > ><br> > ><br> > > root@draid:/usr/src/contrib/bearssl # zfs list -o<br> > > name,type,used,avail,ratio,compression,reservation,refreservation= ,vols<br> > > ize raid-5400-1<br> > ><br> > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE<br> > ><br> > > raid-5400-1 filesystem 18.0T 2.08T 1.36x on none none - <- no = reservation @pool<br> > ><br> > > root@draid:/usr/src/contrib/bearssl #<br> > ><br> > ><br> > ><br> > > root@draid:/usr/src/contrib/bearssl # zfs list -o<br> > > name,type,used,avail,ratio,compression,reservation,refreservation= ,vols<br> > > ize raid-5400-1/esxi-store1<br> > ><br> > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE<br> > ><br> > > raid-5400-1/esxi-store1 volume 16.9T 18.5T 1.78x zstd 15.6T 16.9T= 15.6T <- reservation @dataset<br> > ><br> > ><br> > ><br> > > root@draid:/usr/src/contrib/bearssl # zpool list<br> > ><br> > > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTRO= OT<br> > ><br> > > ccache 9.50G 9.10G 406M - - 88% 95% 1.00x ONLINE -<br> > ><br> > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE - <- Free= does not reflect reservation @pool<br> > ><br> > > tank 18.5T 605G 17.9T - - 0% 3% 1.00x ONLINE -<br> > ><br> > > zfsroot 103G 33.3G 69.7G - - 31% 32% 1.00x ONLINE -<br> > ><br> > > root@draid:/usr/src/contrib/bearssl #<br> > ><br> > ><br> > ><br> > ><br> > ><br> > > CONFIDENTIALITY NOTE: This message is intended only for the use o= f the<br> > > individual or entity to whom it is addressed and may contain<br> > > information that is privileged, confidential, and exempt from<br> > > disclosure under applicable law. If the reader of this message is= not<br> > > the intended recipient, you are hereby notified that any<br> > > dissemination, distribution or copying of this communication is<b= r> > > strictly prohibited. If you have received this transmission in er= ror,<br> > > please notify us by telephone at (502) 212-4000 or notify us at: = PAI,<br> > > Dept. 99,<br> > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245<br> > ><br> > ><br> > ><br> > ><br> > ><br> > > Disclaimer<br> > ><br> > > The information contained in this communication from the sender i= s confidential. It is intended solely for use by the recipient and others a= uthorized to receive it. If you are not the recipient, you are hereby notif= ied that any disclosure, copying, distribution or taking action in relation= of the contents of this information is strictly prohibited and may be unla= wful.<br> > ><br> > > This email has been scanned for viruses and malware, and may have= been automatically archived by Mimecast, a leader in email security and cy= ber resilience. Mimecast integrates email defenses with brand protection, s= ecurity awareness training, web security, compliance and other essential ca= pabilities. Mimecast helps protect large and small organizations from malic= ious activity, human error and technology failure; and to lead the movement= toward building a more resilient world. To find out more, visit our websit= e.<br> ><br> ><br> ><br> ><br> > CONFIDENTIALITY NOTE: This message is intended only for the use<br> > of the individual or entity to whom it is addressed and may<br> > contain information that is privileged, confidential, and<br> > exempt from disclosure under applicable law. If the reader<br> > of this message is not the intended recipient, you are hereby<br> > notified that any dissemination, distribution or copying<br> > of this communication is strictly prohibited. If you have<br> > received this transmission in error, please notify us by<br> > telephone at (502) 212-4000 or notify us at PAI, Dept. 99,<br> > 2101 High Wickham Place, Suite 101, Louisville, KY 40245<br> ><br> ><br> > Disclaimer<br> ><br> > The information contained in this communication from the sender is con= fidential. It is intended solely for use by the recipient and others author= ized to receive it. If you are not the recipient, you are hereby notified t= hat any disclosure, copying, distribution or taking action in relation of t= he contents of this information is strictly prohibited and may be unlawful.= <br> ><br> > This email has been scanned for viruses and malware, and may have been= automatically archived by Mimecast, a leader in email security and cyber r= esilience. Mimecast integrates email defenses with brand protection, securi= ty awareness training, web security, compliance and other essential capabil= ities. Mimecast helps protect large and small organizations from malicious = activity, human error and technology failure; and to lead the movement towa= rd building a more resilient world. To find out more, visit our website.<br= > <br> </blockquote></div> --00000000000081810c05dd313637--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOeNLuoopG9nWe9pX%2BLAHRgqBoygvczsnuE4vAfeQ4uRcgBkKw>