Date: Tue, 28 Jun 2022 17:43:39 -0400 From: Mike Jakubik <mike.jakubik@swiftsmsgateway.com> To: "freebsd-net" <freebsd-net@FreeBSD.org> Cc: "Alexander V. Chernikov" <melifaro@ipfw.ru>, "Dave Cottlehuber" <dch@skunkwerks.at> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Message-ID: <181ac451eba.10ec692122616769.3280827109477949645@swiftsmsgateway.com> In-Reply-To: <9892d8f8-df5d-4e73-ab48-cb5909b4f9c6@www.fastmail.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> <C54B2F28-D02F-4589-82D7-1C84705C2C23@ipfw.ru> <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> <9892d8f8-df5d-4e73-ab48-cb5909b4f9c6@www.fastmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_8280592_2002495593.1656452619962 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, So basically this is my conclusion, if I cpuset iperf on at least the recei= ving end i get great performance. Anything outside of that is random. I've = tried just about every network tuning knob in FreeBSD as well as what Mella= nox recommends in their driver manual, none of these make any significant i= mpact, sometimes they even reduce performance. So I m hoping when this goes= into production the scheduler will be sane enough to do the same, but we s= hall see. Thanks. ---- On Fri, 17 Jun 2022 11:03:04 -0400 Dave Cottlehuber <dch@skunkwerks.at= > wrote --- On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote:=20 > Hi,=20 >=20 > I believe you hit the nail on the head! I am now getting consistent=20 > high speeds, even higher than on Linux! Is this a problem with the=20 > scheduler? Should someone in that area of expertise be made aware of=20 > this? More importantly i guess, would this affect real world=20 > performance, these servers will be running RabbitMQ (it uses quite a=20 > bit of bandwidth) and PostgreSQL w/ replication.=20 =20 pinning cores for unimpeded access is very common for high performance syst= ems. Do this both for the nics and also your apps. Be mindful of the NUMA t= opooogy.=20 =20 You should look into both the erlang scheduler flags for core pinning, and= also ensuring that your erlang processes have unimpeded access to their ow= n cores too. A reasonable approach is to make a simple cowboy or Phoenix ap= p and hammer it with wrk or similar load tool to get a feel for things, and= then profile and tune your own app based on those specific results.=20 =20 For rabbit there is an excellent load testing tool from the pivotal team if= you don=E2=80=99t have suitable load generators yourselves.=20 =20 Tsung is an excellent tool if you put in the work to craft something specif= ic for your use case.=20 =20 Please post back to the list with your specific findings and nic/ tcp tunab= les, these are very helpful for the next person!=20 =20 Dave=20 =20 Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_8280592_2002495593.1656452619962 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>= <meta content=3D"text/html;charset=3DUTF-8" http-equiv=3D"Content-Type"></h= ead><body ><div style=3D"font-family: Verdana, Arial, Helvetica, sans-serif= ; font-size: 10pt;"><div>Hi,<br></div><div><br></div><div>So basically this= is my conclusion, if I cpuset iperf on at least the receiving end i get gr= eat performance. Anything outside of that is random. I've tried just about = every network tuning knob in FreeBSD as well as what Mellanox recommends in= their driver manual, none of these make any significant impact, sometimes = they even reduce performance. So I m hoping when this goes into production = the scheduler will be sane enough to do the same, but we shall see.<br></di= v><div><br></div><div>Thanks.</div><div><br></div><div class=3D"zmail_extra= _hr" style=3D"border-top: 1px solid rgb(204, 204, 204); height: 0px; margin= -top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class=3D= "zmail_extra" data-zbluepencil-ignore=3D"true"><div><br></div><div id=3D"Zm= -_Id_-Sgn1">---- On Fri, 17 Jun 2022 11:03:04 -0400 <b>Dave Cottlehuber <= ;dch@skunkwerks.at></b> wrote ---<br></div><div><br></div><blockquote st= yle=3D"margin: 0px;"><div>On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote= : <br>> Hi, <br>> <br>> I believe you hit the nail on the head! I = am now getting consistent <br>> high speeds, even higher than on Linux! = Is this a problem with the <br>> scheduler? Should someone in that area = of expertise be made aware of <br>> this? More importantly i guess, woul= d this affect real world <br>> performance, these servers will be runnin= g RabbitMQ (it uses quite a <br>> bit of bandwidth) and PostgreSQL w/ re= plication. <br> <br>pinning cores for unimpeded access is very common for h= igh performance systems. Do this both for the nics and also your apps. Be m= indful of the NUMA topooogy. <br> <br>You should look into both the erlang= scheduler flags for core pinning, and also ensuring that your erlang proce= sses have unimpeded access to their own cores too. A reasonable approach is= to make a simple cowboy or Phoenix app and hammer it with wrk or similar l= oad tool to get a feel for things, and then profile and tune your own app b= ased on those specific results. <br> <br>For rabbit there is an excellent l= oad testing tool from the pivotal team if you don=E2=80=99t have suitable l= oad generators yourselves. <br> <br>Tsung is an excellent tool if you put i= n the work to craft something specific for your use case. <br> <br>Please p= ost back to the list with your specific findings and nic/ tcp tunables, the= se are very helpful for the next person! <br> <br>Dave <br> <br></div></blo= ckquote></div><div><br></div><div class=3D"zmail_signature_below"><div id= =3D"Zm-_Id_-Sgn" data-zbluepencil-ignore=3D"true" data-sigid=3D"78860770000= 00096004"><div>Mike Jakubik<br></div><div><a target=3D"_blank" href=3D"http= s://www.swiftsmsgateway.com/">https://www.swiftsmsgateway.com/</a><br></div= ><div><br></div><div><span class=3D"size" style=3D"font-size:10px">Disclaim= er: This e-mail and any attachments are intended only for the use of the ad= dressee(s) and may contain information that is privileged or confidential. = If you are not the intended recipient, or responsible for delivering the in= formation to the intended recipient, you are hereby notified that any disse= mination, distribution, printing or copying of this e-mail and any attachme= nts is strictly prohibited. If this e-mail and any attachments were receive= d in error, please notify the sender by reply e-mail and delete the origina= l message.</span><br></div></div></div><div><br></div></div><br></body></ht= ml> ------=_Part_8280592_2002495593.1656452619962--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?181ac451eba.10ec692122616769.3280827109477949645>