me-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DkzLYAqzkcR60xVohkWb7ttYKWAjfayY6fsL8jkPv00=; b=TkVDHV7jH4dYnvWnLuXLDJkepcnnPh/Y2nyhrvTg9Lezr0P3F2YNBNrvfXa6jDlv44D1zD 9otWMFST9ueN7oezgMG31RGfT3+V0JHFAei9HuWn9FhoxJ34XO7keVA722YVY9v60aozEl 6J7lZS6X/hmIR57nCc6TSnZCqrwjR1o3J4UZIZI5jaQrxoD+tPm0jpgzqjr5F/AzRIaRz+ UkTsGYS3rsgwSAZVeZDNOCuYurzJr+XyqdSW3DZaH74jXYPYAWNtsUd7UJAuryBp9nb3hl e2KwwHxlpo3GoszKn6yCHpZkCj6AJWhozk7kRmm2hvhHmf329KmfdhJckfeD1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1752661779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DkzLYAqzkcR60xVohkWb7ttYKWAjfayY6fsL8jkPv00=; b=j4E+F+PYxauMOI4GOWWRYxUgClvbggjMfzbUnMkzGImmK5JONLBmk/C2wJIcEvvu2TnYKw i0wL/9vgNrHrufwRRtO4SMFfShvbpehk8pOdYSG3a1/ZAYkQH7X2F+b+uj0gxyg9m6tli7 MsgflQ4RmYK7K2g92IPZKxQWkmRDq5KjbBC72Lh/YUL6t8qxzOHnZY7z6EdgNewr0Feqv4 jnTm/d4XNFqSlb/pB/ogVcsLMM2x8ytMk7TiKz1vMJlRrJ8SfFpnpkSgx+n+bYgNhcNL2u bxI8aWLB9/hHBkBSZZ1yyIbMGacavUwYpfd9NRL+lZT6vkcdFJYYG5FkmMf4Qg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1752661779; a=rsa-sha256; cv=none; b=tFmytP561RURNIvXG4LVccoLgZa1M2IGjDBgmGgknwfzW8Rm7qjwOv05N5bRWzFgooog0z GmeUfMnxtK3i/c2LKDIFQAuqIRvBDAoOlh/ltBl1p1BgXG7/NVRI/9sHO3Rl3sWFehg8qw bKO7c6NAAocKkLzjyYELeF6fBp87H6/axxOVyHw2RM7Kheql/ebCjDKDP1fPwU7q8Hij74 mbi2emFdB04TRPJ79fUdIWcCz2SJy1djrxXIbfZFLQrKHgBdbNiJJqBJ+tGw4DYS/Festd b7e9rY+6dWWIMWQvhB20pXob+9aaDe4rDw+O9jw4/K0/x8skvM1E5KnjDleFEg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bhskB6myPz1KNH; Wed, 16 Jul 2025 10:29:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 56GATcvu078932; Wed, 16 Jul 2025 10:29:38 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 56GATcS3078929; Wed, 16 Jul 2025 10:29:38 GMT (envelope-from git) Date: Wed, 16 Jul 2025 10:29:38 GMT Message-Id: <202507161029.56GATcS3078929@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mateusz Piotrowski <0mp@FreeBSD.org> Subject: git: 9388c2887817 - main - dtrace_fbt.4: Document the DTrace fbt provider List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: 0mp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 9388c2887817d7162ebb356b39aa9b4ab67a8c00 Auto-Submitted: auto-generated The branch main has been updated by 0mp: URL: https://cgit.FreeBSD.org/src/commit/?id=9388c2887817d7162ebb356b39aa9b4ab67a8c00 commit 9388c2887817d7162ebb356b39aa9b4ab67a8c00 Author: Mateusz Piotrowski <0mp@FreeBSD.org> AuthorDate: 2025-06-14 19:26:48 +0000 Commit: Mateusz Piotrowski <0mp@FreeBSD.org> CommitDate: 2025-07-16 10:27:52 +0000 dtrace_fbt.4: Document the DTrace fbt provider Reported by: markj Reviewed by: christos, markj (earlier version), ziaee Obtained from: Mark Johnston, DTrace, FreeBSD Journal, May 2014 Obtained from: https://wiki.freebsd.org/DTrace/One-Liners MFC after: 2 weeks Relnotes: yes --- cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 | 3 +- share/man/man4/Makefile | 1 + share/man/man4/dtrace_fbt.4 | 332 +++++++++++++++++++++++++++ share/man/man4/dtrace_kinst.4 | 12 +- 4 files changed, 343 insertions(+), 5 deletions(-) diff --git a/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 b/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 index d44ee5023e78..da8cbd9ffe50 100644 --- a/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 +++ b/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 @@ -20,7 +20,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 14, 2025 +.Dd July 16, 2025 .Dt DTRACE 1 .Os .Sh NAME @@ -1270,6 +1270,7 @@ Invalid command line options or arguments were specified. .Xr dwatch 1 , .Xr dtrace_audit 4 , .Xr dtrace_dtrace 4 , +.Xr dtrace_fbt 4 , .Xr dtrace_io 4 , .Xr dtrace_ip 4 , .Xr dtrace_kinst 4 , diff --git a/share/man/man4/Makefile b/share/man/man4/Makefile index 5e60f00bc09f..fcdbf81dc0fd 100644 --- a/share/man/man4/Makefile +++ b/share/man/man4/Makefile @@ -981,6 +981,7 @@ _ccd.4= ccd.4 .if ${MK_CDDL} != "no" _dtrace_provs= dtrace_audit.4 \ dtrace_dtrace.4 \ + dtrace_fbt.4 \ dtrace_io.4 \ dtrace_ip.4 \ dtrace_kinst.4 \ diff --git a/share/man/man4/dtrace_fbt.4 b/share/man/man4/dtrace_fbt.4 new file mode 100644 index 000000000000..3e35bb8c5bbc --- /dev/null +++ b/share/man/man4/dtrace_fbt.4 @@ -0,0 +1,332 @@ +.\" +.\" SPDX-License-Identifier: BSD-2-Clause +.\" +.\" Copyright (c) 2025 Mateusz Piotrowski <0mp@FreeBSD.org> +.\" +.Dd July 16, 2025 +.Dt DTRACE_FBT 4 +.Os +.Sh NAME +.Nm dtrace_fbt +.Nd a DTrace provider for dynamic kernel tracing based on function boundaries +.Sh SYNOPSIS +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return +.Sh DESCRIPTION +The Function Boundary Tracing +.Pq Nm fbt +provider instruments the entry and return of almost every kernel function +corresponding to an +.Xr elf 5 +symbol in the kernel and loaded kernel modules. +.Pp +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry +fires whenever the +.Ar function +is called. +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return +fires when the +.Ar function +returns. +.Pp +The +.Ar module +in the probe description is either the name of the loaded kernel module +or +.Ql kernel +for functions compiled into the kernel. +.Ss Function Boundary Instrumentation +The +.Nm fbt +will always instrument a function's entry, but +its return will be intsrumented so long as it can find a +.Ql ret +instruction. +.Pp +In some cases, +.Nm fbt +cannot instrument a function's entry and/or return. +Refer to subsection +.Sx Frame Pointer +for more details. +.Ss Probe Arguments +The arguments of the entry probe +.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry +are the arguments of the traced function call. +.Bl -column -offset indent "Entry Probe Argument" "Definition" +.It Sy Entry Probe Argument Ta Sy Definition +.It Fa args[0] Ta Function's first argument, typed +.Pq e.g., Xr malloc 9 Ap s Ft size_t Fa size +.It Fa args[1] Ta Function's second argument, typed +.Pq e.g., Xr malloc 9 Ap s Ft struct malloc_type Fa *type +.It Fa args[2] Ta Function's third argument, typed +.Pq e.g., Xr malloc 9 Ap s Ft int Fa flags +.It Fa ... Ta ... +.El +.Pp +The arguments of the return probe +.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return +are +.Fa args[0] +.Po +the offset of the firing return instruction within the function; +useful to tell apart two different return statements in a single function +.Pc +and +.Fa args[1] +.Pq the return value, if any . +.Bl -column -offset indent "Return Probe Argument" "Definition" +.It Sy Return Probe Argument Ta Sy Definition +.It Fa args[0] Ta Offset of the traced return instruction +.It Fa args[1] Ta Function's return value +.Po e.g., a kernel virtual address if returning from a successful +.Xr malloc 9 +.Pc +.El +.Pp +Subsection +.Sx Example 2 : Getting Details About Probe's Arguments +shows how to get probe's argument count and types directly with +.Xr dtrace 1 +without having to resort to the reading function's source code +or documentation. +.Sh EXAMPLES +.Ss Example 1 : Listing Available FBT Probes +The following example shows how to list all the available +.Nm fbt +probes. +.Bd -literal -offset 2n +# dtrace -l -P fbt + ID PROVIDER MODULE FUNCTION NAME +[...] +31868 fbt kernel hammer_time entry +31869 fbt kernel hammer_time return +[...] +.Ed +.Pp +Since +.Fn hammer_time +is a part of the kernel and not a separate loaded module, the +.Ar module +column displays +.Ql kernel . +.Ss Example 2 : Getting Details About Probe's Arguments +The following example shows how to generate a program stability report of +.Xr malloc 9 Ap s +entry and return probes. +Those reports are useful to view +the probe's number of arguments and their types. +.Bd -literal -offset 2n +# dtrace -l -v -n fbt::malloc:entry +[...] + Argument Types + args[0]: size_t + args[1]: struct malloc_type * + args[2]: int +.Ed +.Pp +The count and types of +.Nm fbt Ns Cm \&::malloc:entry +arguments +match the function signature of +.Xr malloc 9 : +.Va args[0] +is +.Ft size_t , +.Va args[1] +is +.Ft "struct malloc_type *" , +and +.Va "args[2]" +is +.Ft int . +.Bd -literal -offset 2n +# dtrace -l -v -n fbt::malloc:return +[...] + Argument Types + args[0]: int + args[1]: void * +.Ed +.Pp +The +.Cm return +probe reports two arguments and their types: +the return instruction offset +.Pq the usual Ft int +and the function's return value, which in this case is +.Ft void * , +as +.Xr malloc 9 +returns a kernel virtual address. +.Ss Example 3 : Counting Kernel Slab Memory Allocation by Function +.Bd -literal -offset 2n +# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }' +dtrace: description 'fbt::kmem*:entry ' matched 47 probes +^C + kmem_alloc_contig 1 + kmem_alloc_contig_domainset 1 + kmem_cache_reap_active 1 + kmem_alloc_contig_pages 2 + kmem_free 2 + kmem_std_destructor 19 + kmem_std_constructor 26 + kmem_cache_free 151 + kmem_cache_alloc 181 +.Ed +.Ss Example 4 : Counting Kernel Slab Memory Allocation by Calling Function +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\en", @); }' +^C + kernel`contigmalloc+0x33 1 + kernel`free+0xd3 1 + kernel`kmem_alloc_contig+0x29 1 +kernel`kmem_alloc_contig_domainset+0x19a 1 + zfs.ko`arc_reap_cb_check+0x16 1 +.Ed +.Ss Example 5 : Counting Kernel malloc()'s by Calling Function +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\en", @); }' +^C + kernel`devclass_get_devices+0xa8 1 + kernel`sys_ioctl+0xb7 1 + dtrace.ko`dtrace_ioctl+0x15c1 1 + dtrace.ko`dtrace_ioctl+0x972 2 + dtrace.ko`dtrace_dof_create+0x35 2 + kernel`kern_poll_kfds+0x2f0 4 + kernel`kern_poll_kfds+0x28a 19 +.Ed +.Ss Example 6 : Counting Kernel malloc()'s by Kernel Stack Trace +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }' +^C + dtrace.ko`dtrace_dof_create+0x35 + dtrace.ko`dtrace_ioctl+0x827 + kernel`devfs_ioctl+0xd1 + kernel`VOP_IOCTL_APV+0x2a + kernel`vn_ioctl+0xb6 + kernel`devfs_ioctl_f+0x1e + kernel`kern_ioctl+0x286 + kernel`sys_ioctl+0x12f + kernel`amd64_syscall+0x169 + kernel`0xffffffff81092b0b + 2 +.Ed +.Ss Example 7 : Summarizing vmem_alloc()'s by Arena Name and Size Distribution +.Bd -literal -offset 2n +# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }' +^C + + kernel arena dom + value ------------- Distribution ------------- count + 2048 | 0 + 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4 + 8192 |@@@@@@@@@@@@@ 2 + 16384 | 0 +.Ed +.Ss Example 8 : Measuring Total Time Spent Executing a Function +This DTrace script measures the total time spent in +.Fn vm_page* +kernel functions. +The +.Fn quantize +aggregation organizes the measurements into power-of-two buckets, +providing a time distribution in nanoseconds for each function. +.Bd -literal -offset 2n +fbt::vm_page*:entry { + self->start = timestamp; +} + +fbt::vm_page*:return /self->start/ { + @[probefunc] = quantize(timestamp - self->start); + self->start = 0; +} +.Ed +.Sh SEE ALSO +.Xr dtrace 1 , +.Xr dtrace_kinst 4 , +.Xr tracing 7 +.Rs +.%A Brendan Gregg +.%A Jim Mauro +.%B DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD +.%I Prentice Hall +.%P pp. 898\(en903 +.%D 2011 +.%U https://www.brendangregg.com/dtracebook/ +.Re +.Rs +.%B The illumos Dynamic Tracing Guide +.%O Chapter fbt Provider +.%D 2008 +.%U https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt +.Re +.Sh AUTHORS +This manual page was written by +.An Mateusz Piotrowski Aq Mt 0mp@FreeBSD.org . +.Sh CAVEATS +.Ss Stability and Portability +.Nm fbt +probes are by definition tightly coupled to kernel code; if the code underlying +a script changes, the script may fail to run or may produce incorrect results. +Scripts written for one version of +.Fx +might not work on others, +and almost certainly will not work on other operating systems. +.Pp +Individual +.Nm fbt +probes often do not correspond nicely to logical system events. +For example, consider a DTrace script which prints the destination +address of every IP packet as the kernel hands them over +to the network card driver (NIC). +An +.Nm fbt Ns -based +implementation of such a script is a discouragingly difficult task: +it involves instrumenting at least four different functions in different parts +of the IPv4 and IPv6 code. +At the same time, with the +.Xr dtrace_ip 4 +provider the script is a simple one-liner: +.Dl dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}' +.Pp +Make sure to review available +.Xr dtrace 1 +providers first +before implementing a custom script with the +.Nm fbt +provider. +If none of the DTrace providers offer the desired probes, +consider adding new statically-defined tracing probes +.Pq Xr SDT 9 . +.Ss Frame Pointer +Inline functions are not instrumentable by +.Nm fbt +as they lack a frame pointer. +A developer might explicitly disable inlining by adding the +.Ql __noinline +attribute to a function definition, +but of course this requires a recompilation of the kernel. +Building the kernel with +.Fl fno-omit-frame-pointer +is another way of preserving frame pointers. +Note, that sometimes compilers will omit the frame pointer in leaf functions, +even when configured with +.Fl fno-omit-frame-pointer . +.Pp +Function returns via a tail call are also not instrumentable by +.Nm fbt . +As a result, +a function might have an entry probe +and a mix of instrumented and uninstrumentable returns. +.Pp +Use +.Xr dtrace_kinst 4 +to trace arbitrary instructions inside kernel functions +and work around some of the +limitations +of +.Nm fbt . +.Ss Tracing DTrace +The +.Nm fbt +provider cannot attach to functions inside DTrace provider kernel modules. diff --git a/share/man/man4/dtrace_kinst.4 b/share/man/man4/dtrace_kinst.4 index 9debbc1bd106..c2187689749b 100644 --- a/share/man/man4/dtrace_kinst.4 +++ b/share/man/man4/dtrace_kinst.4 @@ -22,7 +22,7 @@ .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" -.Dd February 27, 2023 +.Dd July 16, 2025 .Dt DTRACE_KINST 4 .Os .Sh NAME @@ -43,10 +43,13 @@ creates probes on-demand, meaning it searches for and parses the function's instructions each time .Xr dtrace 1 is run, and not at module load time. -This is in contrast to FBT's load-time parsing, since +This is in contrast to +.Xr dtrace_fbt 4 Ap s +load-time parsing, since .Nm kinst can potentially create thousands of probes for just a single function, instead -of up to two (entry and return) in the case of FBT. +of up to two (entry and return) in the case of +.Xr dtrace_fbt 4 . A result of this is that .Cm dtrace -l -P kinst will not match any probes. @@ -79,7 +82,8 @@ Trace all instructions in # dtrace -n 'kinst::amd64_syscall:' .Ed .Sh SEE ALSO -.Xr dtrace 1 +.Xr dtrace 1 , +.Xr dtrace_fbt 4 .Sh HISTORY The .Nm kinst