From nobody Mon Oct 18 21:32:12 2021 X-Original-To: dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AE0B717FBFC6 for ; Mon, 18 Oct 2021 21:32:23 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HY97p6hwHz4VB9; Mon, 18 Oct 2021 21:32:19 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-wm1-f48.google.com with SMTP id s198-20020a1ca9cf000000b0030d6986ea9fso1146311wme.1; Mon, 18 Oct 2021 14:32:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent :content-language:from:to:subject:content-transfer-encoding; bh=1oxp1I17Qp3JgskrS8/jGW7tq8yFQ2Law3ExoYGnr3Q=; b=4V181jJotpJeDtCVudxCVLxeV4ptB+GxnUVnyiR6hq4Wc/I0cWfcu3ToM5zs8I8IGt MvxkGcRR8sJ8tcpDCerlv4OtizL5BRKQR/wUBjSgyFOWu9qgIM3vRsMBATjrnueS9o6F N4XAAtWthdZDQNugGTq/2XWSKpaukHu17slERGnx521qB7Zt0nb8dSeGdADqHsi4fRVx NdcVq6+jjLHBxwpLERDtboyBsH/XrzaENZ9jHD6RUb3fQJZMkCHLYGwa5/M7DTJL5iGZ om5rQ5cwbK6Nz6snGPgTLPyQnwYDinyYl9SDVaAxSAOhjK0rHSBUm1rtSzD65rvFduMZ yo4Q== X-Gm-Message-State: AOAM532aSVrEWxYH4AHvv3k14KZwfFTPVmrL5M4EFRkvV0KxT832yU9d BIEvOlYQmLrhSzmbQLBayy+sDmPV6X8= X-Google-Smtp-Source: ABdhPJzmRUwhloDIZLqN9OA33ng5SrU+ii9T846vcB3iEBrZ9HdaWGft84sHzu82u8L4ZETVXhX07g== X-Received: by 2002:a05:600c:4f11:: with SMTP id l17mr1512027wmq.77.1634592733310; Mon, 18 Oct 2021 14:32:13 -0700 (PDT) Received: from [192.168.0.15] (ip5f5bd4ef.dynamic.kabel-deutschland.de. [95.91.212.239]) by smtp.gmail.com with ESMTPSA id w26sm515495wmk.34.2021.10.18.14.32.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Oct 2021 14:32:12 -0700 (PDT) Message-ID: Date: Mon, 18 Oct 2021 23:32:12 +0200 List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.2.0 Content-Language: en-US From: Mateusz Piotrowski <0mp@FreeBSD.org> To: dtrace@freebsd.org Subject: Measuring performance impact of tracing a system under load Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4HY97p6hwHz4VB9 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mpp302@gmail.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=mpp302@gmail.com X-Spamd-Result: default: False [1.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[]; NEURAL_SPAM_MEDIUM(0.95)[0.947]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_SHORT(-0.95)[-0.946]; NEURAL_SPAM_LONG(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.48:from]; RECEIVED_SPAMHAUS_PBL(0.00)[95.91.212.239:received]; FORGED_SENDER(0.30)[0mp@FreeBSD.org,mpp302@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.48:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[0mp@FreeBSD.org,mpp302@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello everyone, I would like to measure the overhead / performance impact that of tracing with DTrace has on a system under load. I'm looking for the right way to do it. Obviously, I should not be using DTrace to measure that in this situation. I could use other stats(7) tools but it does not seem. Measuring the impact indirectly sounds like a good idea (e.g., by looking at the throughput between two machines in a test lab as they transfer bytes from A to B in two scenarios: with active tracing and without). Are there any other options? Do you have any insights to share regarding measuring DTrace overhead? I'd be grateful for all suggestions and pointers. Best, Mateusz Piotrowski From nobody Mon Oct 18 23:23:18 2021 X-Original-To: dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 64B5E1803793 for ; Mon, 18 Oct 2021 23:23:26 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from shxd.cx (mail.shxd.cx [64.201.244.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HYCby21KHz3Hf9; Mon, 18 Oct 2021 23:23:26 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from [76.77.180.168] (port=57481 helo=devins-mbp-2.lan) by shxd.cx with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1mcbvP-000CeO-In; Mon, 18 Oct 2021 16:20:11 -0700 Content-Type: text/plain; charset=utf-8 List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: Measuring performance impact of tracing a system under load From: Devin Teske In-Reply-To: Date: Mon, 18 Oct 2021 16:23:18 -0700 Cc: dtrace@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Mateusz Piotrowski <0mp@FreeBSD.org> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4HYCby21KHz3Hf9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N > On Oct 18, 2021, at 2:32 PM, Mateusz Piotrowski <0mp@FreeBSD.org> = wrote: >=20 > Hello everyone, >=20 > I would like to measure the overhead / performance impact that of = tracing with DTrace has on a system under load. I'm looking for the = right way to do it. Obviously, I should not be using DTrace to measure = that in this situation. I could use other stats(7) tools but it does not = seem. Measuring the impact indirectly sounds like a good idea (e.g., by = looking at the throughput between two machines in a test lab as they = transfer bytes from A to B in two scenarios: with active tracing and = without). Are there any other options? Do you have any insights to share = regarding measuring DTrace overhead? >=20 > I'd be grateful for all suggestions and pointers. >=20 I have done this for several sub-systems when I had to =E2=80=94 for = work =E2=80=94 show the progress of bpftrace against DTrace and = formulate plans for sampling strategies based on observed overhead. Running in one shell: dd if=3D/dev/urandom of=3D/tmp/trashfile bs=3D1 count=3D5000000 And another [Linux] shell: bpftrace -e 'tracepoint:syscalls:sys_enter_read { }=E2=80=99 This resulted in a 23% performance hit on the dd while bpftrace was = running. We ran the test 10x and on multiple pieces of hardware, under = multiple Operating Systems, and tried our best to control the testing = environment parameters. The hit on FreeBSD was barely noticeable. I think I calculated it as = less than 6% at its worst. By the way, did you know you can run DTrace on CentOS? cd /etc/yum.repos.d wget http://public-yum.oracle.com/public-yum-ol7.repo yum install -y kernel-uek{,-devel} dtrace-utils # EFI cp /boot/efi/EFI/centos/grub.cfg{,.`date +%s`.bak} grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg # BIOS grubby --set-default-index 0 reboot ASIDE: And it=E2=80=99s actually fun to use and quite powerful. Back to = the question at-hand though =E2=80=A6 I have done similar things with =E2=80=9Cnc=E2=80=9D in place of = =E2=80=9Cdd=E2=80=9D I also suggest checking out =E2=80=9Cdpv=E2=80=9D in FreeBSD base for = rate monitoring with and with-out tracing active to gain insights on = overhead. =E2=80=94=20 Devin=