From nobody Wed Jul 30 09:42:04 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bsS0r2rq8z63QfN; Wed, 30 Jul 2025 09:42:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bsS0r1gBZz47Pl; Wed, 30 Jul 2025 09:42:04 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1753868524; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=nhg8NoDkzLBZ+KW/hubSn4opgu7/r3zT0Yf/BtYIm4U=; b=agJnwJp0w4eWGYB98Lralql8TB9FzxtTXoIPPMa0b3lWmT9l5ktiNtlFSu/Y/NBL/PvArD uI63fhpgFU0suL1MI4mgsWv+tbw7IHUjmxqygvafKbp6SHypmjL66aQbNL1rU4F3A12urC 3yhZi7zhpERu3Q/Mcv7E+/ecNGKsv7SAZ692QMGwgS978p6RcYN7UBRnRELKJEOg9YnFwc 2xvWChpkgbB/VTDesuuFQicq8NoS2SwqIp5qQ6WF8wkwjRHvHGpWmqJELVquR7NbLkffd0 OOX0bcFWa1cYNnD17eJbKiBrHgs92CF5kQGLIk5uD6EwffdrnmBCPqVZNP4sLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1753868524; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=nhg8NoDkzLBZ+KW/hubSn4opgu7/r3zT0Yf/BtYIm4U=; b=TM+wZKdqEAM5RgRso1LtuVvUcrEuSMJyJS57Iw2xoxrOXoE6PXD156N1vCOiMfZj52x2rq 0v2HKCeRbBYN9Ym+Vv4I3BOLrXinf/BA3G+pcCEcUVKMxa/FZbQWP2ivK1sOkcfHrG6BQt ILiHhO8nCFwaAzeB0jeWLdFz9bnHHMrE1vZFjhdGRqc1kMAumisevc9nh1C2dK1AkslIGi Iz9tfzC5kKZxYNRMmAJkAdUDqUdcaMVPpedt2LoPjf64g48AtcQKyb0CeyJh3/VY/B/xf8 uGtTU4B158mi29YwvKuKRBzvk/tq8VmEhBUPIssUtigsQnrZpRuOz/NqFMjeXA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1753868524; a=rsa-sha256; cv=none; b=Vx5yaSLHPSQAsIhxs4IZnU+EqkhA+furtVSLj9/w4B9GjWOwyWTZAzJNtqtlLbuudhUrpk jieoRIOujESt2AH6GLmfKfePFiRwvmtDcQBVzJQTiSlJP4QdFbyYWifahSS1V/xoOaGPNy /3/ZE2ssqJEfsAd81CL32XZjM9X3iZPL+wzifW9zlSQGMol8MaUPbb53vLrHmOPKyVuxQS NP6gayke48LX3xDgMTuDjTf4KdpCVqS3WSJqr8JEVYBy07CGfr67XCBHg0xoEPFd5w/8Nu 7qRufYMpjPCUUD2hSrA7iB5VzkUYYyYpJAUhzqOgEjTtvoIV/75ABAZgqr5CRQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bsS0r1Cy3z19jd; Wed, 30 Jul 2025 09:42:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 56U9g4W5004607; Wed, 30 Jul 2025 09:42:04 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 56U9g4YN004604; Wed, 30 Jul 2025 09:42:04 GMT (envelope-from git) Date: Wed, 30 Jul 2025 09:42:04 GMT Message-Id: <202507300942.56U9g4YN004604@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mateusz Piotrowski <0mp@FreeBSD.org> Subject: git: 2dded8fb5b25 - stable/13 - dtrace_fbt.4: Document the DTrace fbt provider List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: 0mp X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 2dded8fb5b25cb70629d49fecc540ef175a4fff6 Auto-Submitted: auto-generated The branch stable/13 has been updated by 0mp: URL: https://cgit.FreeBSD.org/src/commit/?id=2dded8fb5b25cb70629d49fecc540ef175a4fff6 commit 2dded8fb5b25cb70629d49fecc540ef175a4fff6 Author: Mateusz Piotrowski <0mp@FreeBSD.org> AuthorDate: 2025-06-14 19:26:48 +0000 Commit: Mateusz Piotrowski <0mp@FreeBSD.org> CommitDate: 2025-07-30 09:41:09 +0000 dtrace_fbt.4: Document the DTrace fbt provider Reported by: markj Reviewed by: christos, markj (earlier version), ziaee Obtained from: Mark Johnston, DTrace, FreeBSD Journal, May 2014 Obtained from: https://wiki.freebsd.org/DTrace/One-Liners MFC after: 2 weeks Relnotes: yes (cherry picked from commit 9388c2887817d7162ebb356b39aa9b4ab67a8c00) Do not mention dtrace_kinst(4) as it is not available on FreeBSD 13. --- cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 | 7 + share/man/man4/Makefile | 1 + share/man/man4/dtrace_fbt.4 | 323 +++++++++++++++++++++++++++ 3 files changed, 331 insertions(+) diff --git a/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 b/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 index e09c89cd64d2..9b6eb03226a9 100644 --- a/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 +++ b/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 @@ -18,7 +18,13 @@ .\" CDDL HEADER END .\" Copyright (c) 2006, Sun Microsystems, Inc. All Rights Reserved. .\" +<<<<<<< HEAD .Dd June 14, 2025 +======= +.\" $FreeBSD$ +.\" +.Dd July 16, 2025 +>>>>>>> 9388c2887817 (dtrace_fbt.4: Document the DTrace fbt provider) .Dt DTRACE 1 .Os .Sh NAME @@ -809,6 +815,7 @@ utility first appeared in .Sh SEE ALSO .Xr cpp 1 , .Xr dtrace_audit 4 , +.Xr dtrace_fbt 4 , .Xr dtrace_io 4 , .Xr dtrace_ip 4 , .Xr dtrace_lockstat 4 , diff --git a/share/man/man4/Makefile b/share/man/man4/Makefile index 8802e5fc35d6..5f576c391086 100644 --- a/share/man/man4/Makefile +++ b/share/man/man4/Makefile @@ -960,6 +960,7 @@ _ccd.4= ccd.4 .if ${MK_CDDL} != "no" _dtrace_provs= dtrace_audit.4 \ + dtrace_fbt.4 \ dtrace_io.4 \ dtrace_ip.4 \ dtrace_lockstat.4 \ diff --git a/share/man/man4/dtrace_fbt.4 b/share/man/man4/dtrace_fbt.4 new file mode 100644 index 000000000000..fc55846f2d0b --- /dev/null +++ b/share/man/man4/dtrace_fbt.4 @@ -0,0 +1,323 @@ +.\" +.\" SPDX-License-Identifier: BSD-2-Clause +.\" +.\" Copyright (c) 2025 Mateusz Piotrowski <0mp@FreeBSD.org> +.\" +.Dd July 16, 2025 +.Dt DTRACE_FBT 4 +.Os +.Sh NAME +.Nm dtrace_fbt +.Nd a DTrace provider for dynamic kernel tracing based on function boundaries +.Sh SYNOPSIS +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return +.Sh DESCRIPTION +The Function Boundary Tracing +.Pq Nm fbt +provider instruments the entry and return of almost every kernel function +corresponding to an +.Xr elf 5 +symbol in the kernel and loaded kernel modules. +.Pp +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry +fires whenever the +.Ar function +is called. +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return +fires when the +.Ar function +returns. +.Pp +The +.Ar module +in the probe description is either the name of the loaded kernel module +or +.Ql kernel +for functions compiled into the kernel. +.Ss Function Boundary Instrumentation +The +.Nm fbt +will always instrument a function's entry, but +its return will be intsrumented so long as it can find a +.Ql ret +instruction. +.Pp +In some cases, +.Nm fbt +cannot instrument a function's entry and/or return. +Refer to subsection +.Sx Frame Pointer +for more details. +.Ss Probe Arguments +The arguments of the entry probe +.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry +are the arguments of the traced function call. +.Bl -column -offset indent "Entry Probe Argument" "Definition" +.It Sy Entry Probe Argument Ta Sy Definition +.It Fa args[0] Ta Function's first argument, typed +.Pq e.g., Xr malloc 9 Ap s Ft size_t Fa size +.It Fa args[1] Ta Function's second argument, typed +.Pq e.g., Xr malloc 9 Ap s Ft struct malloc_type Fa *type +.It Fa args[2] Ta Function's third argument, typed +.Pq e.g., Xr malloc 9 Ap s Ft int Fa flags +.It Fa ... Ta ... +.El +.Pp +The arguments of the return probe +.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return +are +.Fa args[0] +.Po +the offset of the firing return instruction within the function; +useful to tell apart two different return statements in a single function +.Pc +and +.Fa args[1] +.Pq the return value, if any . +.Bl -column -offset indent "Return Probe Argument" "Definition" +.It Sy Return Probe Argument Ta Sy Definition +.It Fa args[0] Ta Offset of the traced return instruction +.It Fa args[1] Ta Function's return value +.Po e.g., a kernel virtual address if returning from a successful +.Xr malloc 9 +.Pc +.El +.Pp +Subsection +.Sx Example 2 : Getting Details About Probe's Arguments +shows how to get probe's argument count and types directly with +.Xr dtrace 1 +without having to resort to the reading function's source code +or documentation. +.Sh EXAMPLES +.Ss Example 1 : Listing Available FBT Probes +The following example shows how to list all the available +.Nm fbt +probes. +.Bd -literal -offset 2n +# dtrace -l -P fbt + ID PROVIDER MODULE FUNCTION NAME +[...] +31868 fbt kernel hammer_time entry +31869 fbt kernel hammer_time return +[...] +.Ed +.Pp +Since +.Fn hammer_time +is a part of the kernel and not a separate loaded module, the +.Ar module +column displays +.Ql kernel . +.Ss Example 2 : Getting Details About Probe's Arguments +The following example shows how to generate a program stability report of +.Xr malloc 9 Ap s +entry and return probes. +Those reports are useful to view +the probe's number of arguments and their types. +.Bd -literal -offset 2n +# dtrace -l -v -n fbt::malloc:entry +[...] + Argument Types + args[0]: size_t + args[1]: struct malloc_type * + args[2]: int +.Ed +.Pp +The count and types of +.Nm fbt Ns Cm \&::malloc:entry +arguments +match the function signature of +.Xr malloc 9 : +.Va args[0] +is +.Ft size_t , +.Va args[1] +is +.Ft "struct malloc_type *" , +and +.Va "args[2]" +is +.Ft int . +.Bd -literal -offset 2n +# dtrace -l -v -n fbt::malloc:return +[...] + Argument Types + args[0]: int + args[1]: void * +.Ed +.Pp +The +.Cm return +probe reports two arguments and their types: +the return instruction offset +.Pq the usual Ft int +and the function's return value, which in this case is +.Ft void * , +as +.Xr malloc 9 +returns a kernel virtual address. +.Ss Example 3 : Counting Kernel Slab Memory Allocation by Function +.Bd -literal -offset 2n +# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }' +dtrace: description 'fbt::kmem*:entry ' matched 47 probes +^C + kmem_alloc_contig 1 + kmem_alloc_contig_domainset 1 + kmem_cache_reap_active 1 + kmem_alloc_contig_pages 2 + kmem_free 2 + kmem_std_destructor 19 + kmem_std_constructor 26 + kmem_cache_free 151 + kmem_cache_alloc 181 +.Ed +.Ss Example 4 : Counting Kernel Slab Memory Allocation by Calling Function +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\en", @); }' +^C + kernel`contigmalloc+0x33 1 + kernel`free+0xd3 1 + kernel`kmem_alloc_contig+0x29 1 +kernel`kmem_alloc_contig_domainset+0x19a 1 + zfs.ko`arc_reap_cb_check+0x16 1 +.Ed +.Ss Example 5 : Counting Kernel malloc()'s by Calling Function +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\en", @); }' +^C + kernel`devclass_get_devices+0xa8 1 + kernel`sys_ioctl+0xb7 1 + dtrace.ko`dtrace_ioctl+0x15c1 1 + dtrace.ko`dtrace_ioctl+0x972 2 + dtrace.ko`dtrace_dof_create+0x35 2 + kernel`kern_poll_kfds+0x2f0 4 + kernel`kern_poll_kfds+0x28a 19 +.Ed +.Ss Example 6 : Counting Kernel malloc()'s by Kernel Stack Trace +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }' +^C + dtrace.ko`dtrace_dof_create+0x35 + dtrace.ko`dtrace_ioctl+0x827 + kernel`devfs_ioctl+0xd1 + kernel`VOP_IOCTL_APV+0x2a + kernel`vn_ioctl+0xb6 + kernel`devfs_ioctl_f+0x1e + kernel`kern_ioctl+0x286 + kernel`sys_ioctl+0x12f + kernel`amd64_syscall+0x169 + kernel`0xffffffff81092b0b + 2 +.Ed +.Ss Example 7 : Summarizing vmem_alloc()'s by Arena Name and Size Distribution +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }' +^C + + kernel arena dom + value ------------- Distribution ------------- count + 2048 | 0 + 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4 + 8192 |@@@@@@@@@@@@@ 2 + 16384 | 0 +.Ed +.Ss Example 8 : Measuring Total Time Spent Executing a Function +This DTrace script measures the total time spent in +.Fn vm_page* +kernel functions. +The +.Fn quantize +aggregation organizes the measurements into power-of-two buckets, +providing a time distribution in nanoseconds for each function. +.Bd -literal -offset 2n +fbt::vm_page*:entry { + self->start = timestamp; +} + +fbt::vm_page*:return /self->start/ { + @[probefunc] = quantize(timestamp - self->start); + self->start = 0; +} +.Ed +.Sh SEE ALSO +.Xr dtrace 1 , +.Xr tracing 7 +.Rs +.%A Brendan Gregg +.%A Jim Mauro +.%B DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD +.%I Prentice Hall +.%P pp. 898\(en903 +.%D 2011 +.%U https://www.brendangregg.com/dtracebook/ +.Re +.Rs +.%B The illumos Dynamic Tracing Guide +.%O Chapter fbt Provider +.%D 2008 +.%U https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt +.Re +.Sh AUTHORS +This manual page was written by +.An Mateusz Piotrowski Aq Mt 0mp@FreeBSD.org . +.Sh CAVEATS +.Ss Stability and Portability +.Nm fbt +probes are by definition tightly coupled to kernel code; if the code underlying +a script changes, the script may fail to run or may produce incorrect results. +Scripts written for one version of +.Fx +might not work on others, +and almost certainly will not work on other operating systems. +.Pp +Individual +.Nm fbt +probes often do not correspond nicely to logical system events. +For example, consider a DTrace script which prints the destination +address of every IP packet as the kernel hands them over +to the network card driver (NIC). +An +.Nm fbt Ns -based +implementation of such a script is a discouragingly difficult task: +it involves instrumenting at least four different functions in different parts +of the IPv4 and IPv6 code. +At the same time, with the +.Xr dtrace_ip 4 +provider the script is a simple one-liner: +.Dl dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}' +.Pp +Make sure to review available +.Xr dtrace 1 +providers first +before implementing a custom script with the +.Nm fbt +provider. +If none of the DTrace providers offer the desired probes, +consider adding new statically-defined tracing probes +.Pq Xr SDT 9 . +.Ss Frame Pointer +Inline functions are not instrumentable by +.Nm fbt +as they lack a frame pointer. +A developer might explicitly disable inlining by adding the +.Ql __noinline +attribute to a function definition, +but of course this requires a recompilation of the kernel. +Building the kernel with +.Fl fno-omit-frame-pointer +is another way of preserving frame pointers. +Note, that sometimes compilers will omit the frame pointer in leaf functions, +even when configured with +.Fl fno-omit-frame-pointer . +.Pp +Function returns via a tail call are also not instrumentable by +.Nm fbt . +As a result, +a function might have an entry probe +and a mix of instrumented and uninstrumentable returns. +.Ss Tracing DTrace +The +.Nm fbt +provider cannot attach to functions inside DTrace provider kernel modules.