Date: Tue, 18 Jun 2024 18:47:30 -0600 From: Kyle Taylor <kyle.a.taylor@gmail.com> To: Warner Losh <imp@bsdimp.com> Cc: Yonas Yanfa <yonas.yanfa@gmail.com>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Comments on the latest phoronix benchmark Message-ID: <CAKzM0vsM2yuCV2O6dX6C-uouNpC-bvWy3LS3AqSo8CNbPkZbYg@mail.gmail.com> In-Reply-To: <CANCZdfp1GFiB0oL8P6qaXS7MWZAMuJv1i_OvOTf7dC2xoi6XKg@mail.gmail.com> References: <93872819-faab-461c-ab13-5a3814edb0d6@gmail.com> <CANCZdfp1GFiB0oL8P6qaXS7MWZAMuJv1i_OvOTf7dC2xoi6XKg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000ce1edf061b338b38 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I've been working on a neutral numerical benchmarking interface that uses Python/R. I haven't thought of using it to compare different versions of FreeBSD. I wrote it to compare FreeBSD/Linux for compute intensive workflows. But it would be interesting to compare different versions. I rarely expect the differences between OS versions to be that different, but perhaps that's a mistake. Folks at Klara (that do a lot of impressive benchmarking with FreeBSD) always recommend writing your own tests to capture performance data. Your workloads could be web traffic, databases, compute, etc... but it very likely will not look much like the pre-canned tests that Phoronix puts together. I do look at Larabel's test results on BSD. I follow it with interest, even. But I tend to treat it as early warning testing. I'm usually left trying to figure out a poor result for FreeBSD on my own. In my own CPU/RAM/Filesystem benchmarking workflows on standard installations, FreeBSD and Linux are often very close. Best - Kyle [1] https://klarasystems.com/articles/evaluating-freebsd-current-for-production= -use/ [2] https://the-integral.dev/post/freebsd-for-data-science/ On Tue, Jun 18, 2024, 3:09 PM Warner Losh <imp@bsdimp.com> wrote: > > > On Tue, Jun 18, 2024, 2:08=E2=80=AFPM Yonas Yanfa <yonas.yanfa@gmail.com>= wrote: > >> Hi, >> >> Link: https://www.phoronix.com/review/bsd-linux-threadripper-7980x/3 >> >> The following six benchmarks show FreeBSD 14.1 performed poorly compared >> to other OSes: >> >> [ 4.3x slower ] Stress-NG 0.17.08: Socket Activity -- 3,569 (FreeBSD) vs >> 15,267 (CentOS Stream 9) >> [ 2.9x slower ] Stress-NG 0.17.08: Glibc Qsort Data Sorting -- 779 >> (FreeBSD) vs 2,224 (Ubuntu 24.04 LTS) >> [ 2.2x slower ] Stress-NG 0.17.08: AVX-512 VNNI -- 3,626,943 (FreeBSD) >> vs 8,253,203 (Ubuntu 24.04 LTS) >> [ 1.5x slower ] Stress-NG 0.17.08: CPU Cache -- 2,322,478 (FreeBSD) vs >> 3,557,329 (NetBSD) >> [ 1.5x slower ] Stress-NG 0.17.08: Fused Multiply-Add -- 63,639,465 >> (FreeBSD) vs 96,258,730 (Ubuntu 24.04 LTS) >> [ 1.3x slower ] Stress-NG 0.17.08: Semaphores -- 230,741,240 (FreeBSD) >> vs 313,648,228 (DragonFlyBSD) >> >> Does anyone know why, and how we can improve the numbers? >> > > Stress-ng is not intended to be a benchmark (and says so in its docs) and > does all kinds of special things on Linux only. It has a bunch of stubs o= n > systems that didn=E2=80=99t implement something. It's a deeply flawed. I = believe > this information is in the comments to the article. > > That said, there are speed improvements we can make to things, like our V= M > that other benchmarks do show issues with... but first the benchmarks nee= d > to actually be apples to apples comparisons. > > Warner > > Cheers, >> Yonas >> >> >> --000000000000ce1edf061b338b38 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto">I've been working on a neutral numerical=C2=A0 benchm= arking interface that uses Python/R. I haven't thought of using it to c= ompare different versions of FreeBSD. I wrote it to compare FreeBSD/Linux f= or compute intensive workflows. But it would be interesting to compare diff= erent versions. I rarely expect the differences between OS versions to be t= hat different, but perhaps that's a mistake.<div dir=3D"auto"><br></div= ><div dir=3D"auto">Folks at Klara (that do a lot of impressive benchmarking= with FreeBSD) always recommend writing your own tests to capture performan= ce data. Your workloads could be web traffic, databases, compute, etc... bu= t it very likely will not look much like the pre-canned tests that Phoronix= puts together. I do look at Larabel's test results on BSD. I follow it= with interest, even. But I tend to treat it as early warning testing. I= 9;m usually left trying to figure out a poor result for FreeBSD on my own. = In my own CPU/RAM/Filesystem benchmarking workflows on standard installatio= ns, FreeBSD and Linux are often very close.=C2=A0</div><div dir=3D"auto"><b= r></div><div dir=3D"auto">Best - Kyle</div><div dir=3D"auto"><br></div><div= dir=3D"auto"><br></div><div dir=3D"auto">[1]=C2=A0<a href=3D"https://klara= systems.com/articles/evaluating-freebsd-current-for-production-use/">https:= //klarasystems.com/articles/evaluating-freebsd-current-for-production-use/<= /a></div><div dir=3D"auto">[2] <a href=3D"https://the-integral.dev/post/fre= ebsd-for-data-science/">https://the-integral.dev/post/freebsd-for-data-scie= nce/</a></div><div dir=3D"auto"><div dir=3D"auto"><br></div><div dir=3D"aut= o"><br></div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" cl= ass=3D"gmail_attr">On Tue, Jun 18, 2024, 3:09 PM Warner Losh <<a href=3D= "mailto:imp@bsdimp.com">imp@bsdimp.com</a>> wrote:<br></div><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid= ;padding-left:1ex"><div dir=3D"auto"><div><br><br><div class=3D"gmail_quote= "><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Jun 18, 2024, 2:08=E2=80=AF= PM Yonas Yanfa <<a href=3D"mailto:yonas.yanfa@gmail.com" target=3D"_blan= k" rel=3D"noreferrer">yonas.yanfa@gmail.com</a>> wrote:<br></div><blockq= uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex">Hi,<br> <br> Link: <a href=3D"https://www.phoronix.com/review/bsd-linux-threadripper-798= 0x/3" rel=3D"noreferrer noreferrer noreferrer" target=3D"_blank">https://ww= w.phoronix.com/review/bsd-linux-threadripper-7980x/3</a><br> <br> The following six benchmarks show FreeBSD 14.1 performed poorly compared <b= r> to other OSes:<br> <br> [ 4.3x slower ] Stress-NG 0.17.08: Socket Activity -- 3,569 (FreeBSD) vs <b= r> 15,267 (CentOS Stream 9)<br> [ 2.9x slower ] Stress-NG 0.17.08: Glibc Qsort Data Sorting -- 779 <br> (FreeBSD) vs 2,224 (Ubuntu 24.04 LTS)<br> [ 2.2x slower ] Stress-NG 0.17.08: AVX-512 VNNI -- 3,626,943 (FreeBSD) <br> vs 8,253,203 (Ubuntu 24.04 LTS)<br> [ 1.5x slower ] Stress-NG 0.17.08: CPU Cache -- 2,322,478 (FreeBSD) vs <br> 3,557,329 (NetBSD)<br> [ 1.5x slower ] Stress-NG 0.17.08: Fused Multiply-Add -- 63,639,465 <br> (FreeBSD) vs 96,258,730 (Ubuntu 24.04 LTS)<br> [ 1.3x slower ] Stress-NG 0.17.08: Semaphores -- 230,741,240 (FreeBSD) <br> vs 313,648,228 (DragonFlyBSD)<br> <br> Does anyone know why, and how we can improve the numbers?<br></blockquote><= /div></div><div dir=3D"auto"><br></div><div dir=3D"auto">Stress-ng is not i= ntended to be a benchmark (and says so in its docs) and does all kinds of s= pecial things on Linux only. It has a bunch of stubs on systems that didn= =E2=80=99t implement something. It's a deeply flawed. I believe this in= formation is in the comments to the article.</div><div dir=3D"auto"><br></d= iv><div dir=3D"auto">That said, there are speed improvements we can make to= things, like our VM that other benchmarks do show issues with... but first= the benchmarks need to actually be apples to apples comparisons.</div><div= dir=3D"auto"><br></div><div dir=3D"auto">Warner</div><div dir=3D"auto"><br= ></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=3D"gm= ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le= ft:1ex"> Cheers,<br> Yonas<br> <br> <br> </blockquote></div></div></div> </blockquote></div> --000000000000ce1edf061b338b38--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKzM0vsM2yuCV2O6dX6C-uouNpC-bvWy3LS3AqSo8CNbPkZbYg>