Date: Wed, 06 Apr 2022 18:34:56 +0200
From: egoitz@ramattack.net
To: Stefan Esser <se@freebsd.org>
Cc: freebsd-fs@freebsd.org, freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Rainer Duffner <rainer@ultra-secure.de>
Subject: Re: {* 05.00 *}Re: Desperate with 870 QVO and ZFS
Message-ID: <ce51660b5f83f92aa9772d764ae12dff@ramattack.net>
In-Reply-To: <dd9a55ac-053d-7802-169d-04c95c045ed2@FreeBSD.org>
References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> <665236B1-8F61-4B0E-BD9B-7B501B8BD617@ultra-secure.de> <0ef282aee34b441f1991334e2edbcaec@ramattack.net> <dd9a55ac-053d-7802-169d-04c95c045ed2@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--=_9e84ea9eb28b05e81541398ce76d2803 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Hi Stefan! Thank you so much for your answer!!. I do answer below in green bold for instance... for a better distinction.... Very thankful for all your comments Stefan!!! :) :) :) Cheers!! El 2022-04-06 17:43, Stefan Esser escribió: > ATENCION > ATENCION > ATENCION!!! Este correo se ha enviado desde fuera de la organizacion. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro. > > Am 06.04.22 um 16:36 schrieb egoitz@ramattack.net: > >> Hi Rainer! >> >> Thank you so much for your help :) :) >> >> Well I assume they are in a datacenter and should not be a power outage.... >> >> About dataset size... yes... our ones are big... they can be 3-4 TB easily each >> dataset..... >> >> We bought them, because as they are for mailboxes and mailboxes grow and >> grow.... for having space for hosting them... > > Which mailbox format (e.g. mbox, maildir, ...) do you use? > > I'M RUNNING CYRUS IMAP SO SORT OF MAILDIR... TOO MANY LITTLE FILES NORMALLY..... SOMETIMES DIRECTORIES WITH TONS OF LITTLE FILES.... > >> We knew they had some speed issues, but those speed issues, we thought (as >> Samsung explains in the QVO site) they started after exceeding the speeding >> buffer this disks have. We though that meanwhile you didn't exceed it's >> capacity (the capacity of the speeding buffer) no speed problem arises. Perhaps >> we were wrong?. > > These drives are meant for small loads in a typical PC use case, > i.e. some installations of software in the few GB range, else only > files of a few MB being written, perhaps an import of media files > that range from tens to a few hundred MB at a time, but less often > than once a day. > > WE MOVE, YOU KNOW... LOTS OF LITTLE FILES... AND LOT'S OF DIFFERENT CONCURRENT MODIFICATIONS BY 1500-2000 CONCURRENT IMAP CONNECTIONS WE HAVE... > > As the SSD fills, the space available for the single level write > cache gets smaller > > THE SINGLE LEVEL WRITE CACHE IS THE CACHE THESE SSD DRIVERS HAVE, FOR COMPENSATING THE SPEED ISSUES THEY HAVE DUE TO USING QLC MEMORY?. DO YOU REFER TO THAT?. SORRY I DON'T UNDERSTAND WELL THIS PARAGRAPH. > > (on many SSDs, I have no numbers for this > particular device), and thus the amount of data that can be > written at single cell speed shrinks as the SSD gets full. > > I have just looked up the size of the SLC cache, it is specified > to be 78 GB for the empty SSD, 6 GB when it is full (for the 2 TB > version, smaller models will have a smaller SLC cache). > > ASSUMING YOU WERE TALKING ABOUT THE CACHE FOR COMPENSATING SPEED WE PREVIOUSLY COMMENTED, I SHOULD SAY THESE ARE THE 870 QVO BUT THE 8TB VERSION. SO THEY SHOULD HAVE THE BIGGEST CACHE FOR COMPENSATING THE SPEED ISSUES... > > But after writing those few GB at a speed of some 500 MB/s (i.e. > after 12 to 150 seconds), the drive will need several minutes to > transfer those writes to the quad-level cells, and will operate > at a fraction of the nominal performance during that time. > (QLC writes max out at 80 MB/s for the 1 TB model, 160 MB/s for the > 2 TB model.) > > WELL WE ARE IN THE 8TB MODEL. I THINK I HAVE UNDERSTOOD WHAT YOU WROTE IN PREVIOUS PARAGRAPH. YOU SAID THEY CAN BE FAST BUT NOT CONSTANTLY, BECAUSE LATER THEY HAVE TO WRITE ALL THAT TO THEIR PERPETUAL STORAGE FROM THE CACHE. AND THAT'S SLOW. AM I WRONG?. EVEN IN THE 8TB MODEL YOU THINK STEFAN?. > > THE MAIN PROBLEM WE ARE FACING IS THAT IN SOME PEAK MOMENTS, WHEN THE MACHINE SERVES CONNECTIONS FOR ALL THE INSTANCES IT HAS, AND ONLY AS SAID IN SOME PEAK MOMENTS... LIKE THE 09AM OR THE 11AM.... IT SEEMS THE MACHINE BECOMES SLOWER... AND LIKE IF THE DISKS WEREN'T ABLE TO SERVE ALL THEY HAVE TO SERVE.... IN THESE MOMENTS, NO BIG FILES ARE MOVED... BUT AS WE HAVE 1800-2000 CONCURRENT IMAP CONNECTIONS... NORMALLY THEY ARE DOING EACH ONE... LITTLE CHANGES IN THEIR MAILBOX. DO YOU THINK PERHAPS THIS DISKS THEN ARE NOT APPROPRIATE FOR THIS KIND OF USAGE?- > > And cheap SSDs often have no RAM cache (not checked, but I'd be > surprised if the QVO had one) and thus cannot keep bookkeeping date > in such a cache, further limiting the performance under load. > > THIS BROCHURE (HTTPS://SEMICONDUCTOR.SAMSUNG.COM/RESOURCES/BROCHURE/870_SERIES_BROCHURE.PDF AND THE DATASHEET HTTPS://SEMICONDUCTOR.SAMSUNG.COM/RESOURCES/DATA-SHEET/SAMSUNG_SSD_870_QVO_DATA_SHEET_REV1.1.PDF) SAIS IF I HAVE READ PROPERLY, THE 8TB DRIVE HAS 8GB OF RAM?. I ASSUME THAT IS WHAT THEY CALL THE TURBO WRITE CACHE?. > > And the resilience (max. amount of data written over its lifetime) > is also quite low - I hope those drives are used in some kind of > RAID configuration. > > YEP WE USE RAIDZ-2 > > The 870 QVO is specified for 370 full capacity > writes, i.e. 370 TB for the 1 TB model. That's still a few hundred > GB a day - but only if the write amplification stays in a reasonable > range ... > > WELL YES... 2880TB IN OUR CASE....NOT BAD.. ISN'T IT? --=_9e84ea9eb28b05e81541398ce76d2803 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset= =3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family: Verdana,Gen= eva,sans-serif'> <p>Hi Stefan!</p> <p><br /></p> <p>Thank you so much for your answer!!. I do answer below in green bold for= instance... for a better distinction....</p> <p><br /></p> <p>Very thankful for all your comments Stefan!!! :) :) :)</p> <p><br /></p> <p>Cheers!!</p> <div> </div> <p><br /></p> <p>El 2022-04-06 17:43, Stefan Esser escribió:</p> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ig= nored --> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= ATENCION<br /> ATENCION<br /> ATENCION!!! Este correo se ha enviado desde f= uera de la organizacion. No pinche en los enlaces ni abra los adjuntos a no= ser que reconozca el remitente y sepa que el contenido es seguro.<br /> <b= r /> Am 06.04.22 um 16:36 schrieb egoitz@ramattack.net: <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0">Hi Rainer!<br /> <br /> Thank you so much for your hel= p :) :)<br /> <br /> Well I assume they are in a datacenter and should not = be a power outage....<br /> <br /> About dataset size... yes... our ones ar= e big... they can be 3-4 TB easily each<br /> dataset.....<br /> <br /> We = bought them, because as they are for mailboxes and mailboxes grow and<br />= grow.... for having space for hosting them...</blockquote> <br /> Which mailbox format (e.g. mbox, maildir, ...) do you use?</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">I'm running Cyrus imap so sort of M= aildir... too many little files normally..... Sometimes directories with to= ns of little files....</span></strong><br /> <br /> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0">We knew they had some speed issues, but those speed is= sues, we thought (as<br /> Samsung explains in the QVO site) they started a= fter exceeding the speeding<br /> buffer this disks have. We though that me= anwhile you didn't exceed it's<br /> capacity (the capacity of the speeding= buffer) no speed problem arises. Perhaps<br /> we were wrong?.</blockquote= > <br /> These drives are meant for small loads in a typical PC use case,<br = /> i.e. some installations of software in the few GB range, else only<br />= files of a few MB being written, perhaps an import of media files<br /> th= at range from tens to a few hundred MB at a time, but less often<br /> than= once a day.</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">We move, you know... lots of little= files... and lot's of different concurrent modifications by 1500-2000 conc= urrent imap connections we have...</span></strong></div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <br /> As the SSD fills, the space available for the single level write<br = /> cache gets smaller</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">The single level write cache is the= cache these ssd drivers have, for compensating the speed issues they have = due to using qlc memory?. Do you refer to that?. Sorry I don't understand w= ell this paragraph.</span></strong></div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= (on many SSDs, I have no numbers for this<br /> particular device), and thu= s the amount of data that can be<br /> written at single cell speed shrinks= as the SSD gets full.</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <br /> <br /> I have just looked up the size of the SLC cache, it is specif= ied<br /> to be 78 GB for the empty SSD, 6 GB when it is full (for the 2 TB= <br /> version, smaller models will have a smaller SLC cache).</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">Assuming you were talking about the= cache for compensating speed we previously commented, I should say these a= re the 870 QVO but the 8TB version. So they should have the biggest cache f= or compensating the speed issues...</span></strong></div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <br /> <br /> But after writing those few GB at a speed of some 500 MB/s (i= =2Ee.<br /> after 12 to 150 seconds), the drive will need several minutes t= o<br /> transfer those writes to the quad-level cells, and will operate<br = /> at a fraction of the nominal performance during that time.<br /> (QLC wr= ites max out at 80 MB/s for the 1 TB model, 160 MB/s for the<br /> 2 TB mod= el.)</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">Well we are in the 8TB model. I thi= nk I have understood what you wrote in previous paragraph. You said they ca= n be fast but not constantly, because later they have to write all that to = their perpetual storage from the cache. And that's slow. Am I wrong?. Even = in the 8TB model you think Stefan?.</span></strong></div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <span style=3D"color: #008000;"><strong>The main problem we are facing is t= hat in some peak moments, when the machine serves connections for all the i= nstances it has, and only as said in some peak moments... like the 09am or = the 11am.... it seems the machine becomes slower... and like if the disks w= eren't able to serve all they have to serve.... In these moments, no big fi= les are moved... but as we have 1800-2000 concurrent imap connections... no= rmally they are doing each one... little changes in their mailbox. Do you t= hink perhaps this disks then are not appropriate for this kind of usage?-</= strong></span><br /> <br /> And cheap SSDs often have no RAM cache (not che= cked, but I'd be<br /> surprised if the QVO had one) and thus cannot keep b= ookkeeping date<br /> in such a cache, further limiting the performance und= er load.</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">This brochure (<a style=3D"color: #= 008000;" href=3D"https://semiconductor.samsung.com/resources/brochure/870_S= eries_Brochure.pdf">https://semiconductor.samsung.com/resources/brochure/87= 0_Series_Brochure.pdf</a> and the datasheet https://semiconductor.samsung= =2Ecom/resources/data-sheet/Samsung_SSD_870_QVO_Data_Sheet_Rev1.1.pdf) sais= if I have read properly, the 8TB drive has 8GB of ram?. I assume that is w= hat they call the turbo write cache?.</span></strong><br /> <br /> And the = resilience (max. amount of data written over its lifetime)<br /> is also qu= ite low - I hope those drives are used in some kind of<br /> RAID configura= tion.</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">Yep we use raidz-2</span></strong><= /div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= The 870 QVO is specified for 370 full capacity<br /> writes, i.e. 370 TB fo= r the 1 TB model. That's still a few hundred<br /> GB a day - but only if t= he write amplification stays in a reasonable<br /> range ...</div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= </div> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <strong><span style=3D"color: #008000;">Well yes... 2880TB in our case...= =2Enot bad.. isn't it?</span></strong></div> </blockquote> </body></html> --=_9e84ea9eb28b05e81541398ce76d2803--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ce51660b5f83f92aa9772d764ae12dff>