Date: Fri, 19 Apr 2024 16:23:51 -0600 From: Alan Somers <asomers@freebsd.org> To: FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Stressing malloc(9) Message-ID: <CAOtMX2jeDHS15bGgzD89AOAd1SzS_=FikorkCdv9-eAxCZ2P5w@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
TLDR; How can I create a workload that causes malloc(9)'s performance to plummet? Background: I recently witnessed a performance problem on a production server. Overall throughput dropped by over 30x. dtrace showed that 60% of the CPU time was dominated by lock_delay as called by three functions: printf (via ctl_worker_thread), g_eli_alloc_data, and g_eli_write_done. One thing those three have in common is that they all use malloc(9). Fixing the problem was as simple as telling CTL to stop printing so many warnings, by tuning kern.cam.ctl.time_io_secs=100000. But even with CTL quieted, dtrace still reports ~6% of the CPU cycles in lock_delay via g_eli_alloc_data. So I believe that malloc is limiting geli's performance. I would like to try replacing it with uma(9). But on a non-production server, none of my benchmark workloads causes g_eli_alloc_data to break a sweat. I can't get its CPU consumption to rise higher than 0.5%. And that's using the smallest sector size and block size that I can. So my question is: does anybody have a program that can really stress malloc(9)? I'd like to run it in parallel with my geli benchmarks to see how much it interferes. -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jeDHS15bGgzD89AOAd1SzS_=FikorkCdv9-eAxCZ2P5w>