Date: Wed, 06 Apr 2022 13:26:24 +0200 From: egoitz@ramattack.net To: Freebsd hackers <freebsd-hackers@freebsd.org> Cc: owner-freebsd-hackers@freebsd.org Subject: Re: Desperate with 870 QVO and ZFS Message-ID: <d2a8a7d2663b4d90e873bfcf95fd2e03@ramattack.net> In-Reply-To: <6cf6c03c5a4aa8128575ec4e2f70b168@ramattack.net> References: <6cf6c03c5a4aa8128575ec4e2f70b168@ramattack.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--=_7b94aa3284999df1ebddce202b1f53e2 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 The most extrange thing is... When machine boots ARC is in 40 value of GB used (for instance), but later decreases to 20GB (and this is not an example... is exact) in all my servers.... it's like if the ARC metadata which is more or less 17GB would limite the whole ARC..... With the traffic of this machines, it should I suppose the ARC should be larger than it is... and ARC in loader.conf is limited to 64GB (the half the ram this machines have) El 2022-04-06 13:18, egoitz@ramattack.net escribió: > ATENCION: Este correo se ha enviado desde fuera de la organización. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro. > > Good morning, > > I write this post with the expectation that perhaps someone could help me > > I am running some mail servers with FreeBSD and ZFS. They use 870 QVO (not EVO or other Samsung SSD disks) disks as storage. They can easily have from 1500 to 2000 concurrent connections. The machines have 128GB of ram and the CPU is almost absolutely idle. The disk IO is normally at 30 or 40% percent at most. > > The problem I'm facing is that they could be running just fine and suddenly at some peak hour, the IO goes to 60 or 70% and the machine becomes extremely slow. ZFS is all by default, except the sync parameter which is set disabled. Apart from that the ARC is limited to 64GB. But even this is extremely odd. The used ARC is near 20GB. I have seen, that meta cache in arc is very near to the limit that FreeBSD automatically sets depending on the size of the ARC you set. It seems that almost all ARC is used by meta cache. I have seen this effect in all my mail servers with this hardware and software config. > > I do attach a zfs-stats output, but from now that the servers are not so loaded as described. I do explain. I run a couple of Cyrus instances in these servers. One as master, one as slave on each server. The commented situation from above, happens when both Cyrus instances become master, so when we are using two Cyrus instances giving service in the same machine. For avoiding issues, know we have balanced and we have a master and a slave in each server. You know, a slave instance has almost no io and only a single connection for replication. So the zfs-stats output is from now we have let's say half of load in each server, because they have one master and one slave instance. > > As said before, when I place two masters in same server, perhaps all day works, but just at 11:00 am (for example) the IO goes to 60% (it doesn't increase) but it seems like if the IO where not being able to be served, let's say more than a limit. More than a concrete io limit (I'd say 60%). > > I don't really know if, perhaps the QVO technology could be the guilty here.... because... they say are desktop computers disks... but later... I have get a nice performance when copying for instance mailboxes from five to five.... I can flood a gigabit interface when copying mailboxes between servers from five to five.... they seem to perform.... > > Could anyone please shed us some light in this issue?. I don't really know what to think. > > Best regards, > > ATENCION: Este correo se ha enviado desde fuera de la organización. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro. --=_7b94aa3284999df1ebddce202b1f53e2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset= =3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family: Verdana,Gen= eva,sans-serif'> <p data-xf-p=3D"1">The most extrange thing is... When machine boots ARC is = in 40 value of GB used (for instance), but later decreases to 20GB (and thi= s is not an example... is exact) in all my servers.... it's like if the ARC= metadata which is more or less 17GB would limite the whole ARC.....</p> <p data-xf-p=3D"1"> </p> <p data-xf-p=3D"1">With the traffic of this machines, it should I suppose t= he ARC should be larger than it is... and ARC in loader.conf is limited to = 64GB (the half the ram this machines have)</p> <div> </div> <p><br /></p> <p>El 2022-04-06 13:18, egoitz@ramattack.net escribió:</p> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0"><!-- html ignored --><br /> <table style=3D"background-color: yellow; border: 1px solid red; font-famil= y: Arial; font-size: 12px; font-weight: bold;" width=3D"100%"> <tbody> <tr> <td><span style=3D"color: red;">ATENCION:</span> Este correo se ha enviado = desde fuera de la organización. No pinche en los enlaces ni abra los= adjuntos a no ser que reconozca el remitente y sepa que el contenido es se= guro.</td> </tr> </tbody> </table> <br /> <!-- meta ignored --> <div class=3D"bbWrapper">Good morning,<br /> <br /> I write this post with = the expectation that perhaps someone could help me <img class=3D"smilie smi= lie--sprite smilie--sprite1" title=3D"Smile :)" alt=3D":)" /><br /> <br = /> I am running some mail servers with FreeBSD and ZFS. They use 870 QVO (n= ot EVO or other Samsung SSD disks) disks as storage. They can easily have f= rom 1500 to 2000 concurrent connections. The machines have 128GB of ram and= the CPU is almost absolutely idle. The disk IO is normally at 30 or 40% pe= rcent at most.<br /> <br /> The problem I'm facing is that they could be ru= nning just fine and suddenly at some peak hour, the IO goes to 60 or 70% an= d the machine becomes extremely slow. ZFS is all by default, except the syn= c parameter which is set disabled. Apart from that the ARC is limited to 64= GB. But even this is extremely odd. The used ARC is near 20GB. I have seen,= that meta cache in arc is very near to the limit that FreeBSD automaticall= y sets depending on the size of the ARC you set. It seems that almost all A= RC is used by meta cache. I have seen this effect in all my mail servers wi= th this hardware and software config.<br /> <br /> I do attach a zfs-stats = output, but from now that the servers are not so loaded as described. I do = explain. I run a couple of Cyrus instances in these servers. One as master,= one as slave on each server. The commented situation from above, happens w= hen both Cyrus instances become master, so when we are using two Cyrus inst= ances giving service in the same machine. For avoiding issues, know we have= balanced and we have a master and a slave in each server. You know, a slav= e instance has almost no io and only a single connection for replication. S= o the zfs-stats output is from now we have let's say half of load in each s= erver, because they have one master and one slave instance.<br /> <br /> As= said before, when I place two masters in same server, perhaps all day work= s, but just at 11:00 am (for example) the IO goes to 60% (it doesn't increa= se) but it seems like if the IO where not being able to be served, let's sa= y more than a limit. More than a concrete io limit (I'd say 60%).<br /> <br= /> I don't really know if, perhaps the QVO technology could be the guilty = here.... because... they say are desktop computers disks... but later... I = have get a nice performance when copying for instance mailboxes from five t= o five.... I can flood a gigabit interface when copying mailboxes between s= ervers from five to five.... they seem to perform....<br /> <br /> Could an= yone please shed us some light in this issue?. I don't really know what to = think.<br /> <br /> Best regards,</div> <div class=3D"js-selectToQuoteEnd"> </div> <p><br /></p> <br /><br /> <table style=3D"background-color: yellow; border: 1px solid red; font-famil= y: Arial; font-size: 12px; font-weight: bold;" width=3D"100%"> <tbody> <tr> <td><span style=3D"color: red;">ATENCION:</span> Este correo se ha enviado = desde fuera de la organización. No pinche en los enlaces ni abra los= adjuntos a no ser que reconozca el remitente y sepa que el contenido es se= guro.</td> </tr> </tbody> </table> </blockquote> </body></html> --=_7b94aa3284999df1ebddce202b1f53e2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d2a8a7d2663b4d90e873bfcf95fd2e03>