From nobody Mon Oct 18 23:23:18 2021 X-Original-To: dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 64B5E1803793 for ; Mon, 18 Oct 2021 23:23:26 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from shxd.cx (mail.shxd.cx [64.201.244.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HYCby21KHz3Hf9; Mon, 18 Oct 2021 23:23:26 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from [76.77.180.168] (port=57481 helo=devins-mbp-2.lan) by shxd.cx with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1mcbvP-000CeO-In; Mon, 18 Oct 2021 16:20:11 -0700 Content-Type: text/plain; charset=utf-8 List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: Measuring performance impact of tracing a system under load From: Devin Teske In-Reply-To: Date: Mon, 18 Oct 2021 16:23:18 -0700 Cc: dtrace@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Mateusz Piotrowski <0mp@FreeBSD.org> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4HYCby21KHz3Hf9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N > On Oct 18, 2021, at 2:32 PM, Mateusz Piotrowski <0mp@FreeBSD.org> = wrote: >=20 > Hello everyone, >=20 > I would like to measure the overhead / performance impact that of = tracing with DTrace has on a system under load. I'm looking for the = right way to do it. Obviously, I should not be using DTrace to measure = that in this situation. I could use other stats(7) tools but it does not = seem. Measuring the impact indirectly sounds like a good idea (e.g., by = looking at the throughput between two machines in a test lab as they = transfer bytes from A to B in two scenarios: with active tracing and = without). Are there any other options? Do you have any insights to share = regarding measuring DTrace overhead? >=20 > I'd be grateful for all suggestions and pointers. >=20 I have done this for several sub-systems when I had to =E2=80=94 for = work =E2=80=94 show the progress of bpftrace against DTrace and = formulate plans for sampling strategies based on observed overhead. Running in one shell: dd if=3D/dev/urandom of=3D/tmp/trashfile bs=3D1 count=3D5000000 And another [Linux] shell: bpftrace -e 'tracepoint:syscalls:sys_enter_read { }=E2=80=99 This resulted in a 23% performance hit on the dd while bpftrace was = running. We ran the test 10x and on multiple pieces of hardware, under = multiple Operating Systems, and tried our best to control the testing = environment parameters. The hit on FreeBSD was barely noticeable. I think I calculated it as = less than 6% at its worst. By the way, did you know you can run DTrace on CentOS? cd /etc/yum.repos.d wget http://public-yum.oracle.com/public-yum-ol7.repo yum install -y kernel-uek{,-devel} dtrace-utils # EFI cp /boot/efi/EFI/centos/grub.cfg{,.`date +%s`.bak} grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg # BIOS grubby --set-default-index 0 reboot ASIDE: And it=E2=80=99s actually fun to use and quite powerful. Back to = the question at-hand though =E2=80=A6 I have done similar things with =E2=80=9Cnc=E2=80=9D in place of = =E2=80=9Cdd=E2=80=9D I also suggest checking out =E2=80=9Cdpv=E2=80=9D in FreeBSD base for = rate monitoring with and with-out tracing active to gain insights on = overhead. =E2=80=94=20 Devin=