Date: Wed, 30 Jul 2025 14:30:54 +0000 (UTC) From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> To: Mateusz Piotrowski <0mp@FreeBSD.org> Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org Subject: Re: git: 0c91fa982437 - stable/14 - dtrace_fbt.4: Document the DTrace fbt provider Message-ID: <sroo78rp-5r3r-1qo3-84on-r316rq9qo582@yvfgf.mnoonqbm.arg> In-Reply-To: <202507300936.56U9aujh087527@gitrepo.freebsd.org> References: <202507300936.56U9aujh087527@gitrepo.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 30 Jul 2025, Mateusz Piotrowski wrote: > The branch stable/14 has been updated by 0mp: > > URL: https://cgit.FreeBSD.org/src/commit/?id=0c91fa982437417e09218ad8c2428aff29d9c5bd > > commit 0c91fa982437417e09218ad8c2428aff29d9c5bd > Author: Mateusz Piotrowski <0mp@FreeBSD.org> > AuthorDate: 2025-06-14 19:26:48 +0000 > Commit: Mateusz Piotrowski <0mp@FreeBSD.org> > CommitDate: 2025-07-30 09:36:44 +0000 > > dtrace_fbt.4: Document the DTrace fbt provider > > Reported by: markj > Reviewed by: christos, markj (earlier version), ziaee > Obtained from: Mark Johnston, DTrace, FreeBSD Journal, May 2014 > Obtained from: https://wiki.freebsd.org/DTrace/One-Liners > MFC after: 2 weeks > Relnotes: yes > > (cherry picked from commit 9388c2887817d7162ebb356b39aa9b4ab67a8c00) > --- > cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 | 5 + > share/man/man4/Makefile | 1 + > share/man/man4/dtrace_fbt.4 | 332 +++++++++++++++++++++++++++ > share/man/man4/dtrace_kinst.4 | 12 +- > 4 files changed, 346 insertions(+), 4 deletions(-) > > diff --git a/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 b/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 > index 1836707d72df..9cb6e9dd276f 100644 > --- a/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 > +++ b/cddl/contrib/opensolaris/cmd/dtrace/dtrace.1 > @@ -20,7 +20,11 @@ > .\" > .\" $FreeBSD$ > .\" > +<<<<<<< HEAD > .Dd June 14, 2025 > +======= > +.Dd July 16, 2025 > +>>>>>>> 9388c2887817 (dtrace_fbt.4: Document the DTrace fbt provider) This merge conflict wasn't resolved? > .Dt DTRACE 1 > .Os > .Sh NAME > @@ -1222,6 +1226,7 @@ Invalid command line options or arguments were specified. > .Sh SEE ALSO > .Xr cpp 1 , > .Xr dtrace_audit 4 , > +.Xr dtrace_fbt 4 , > .Xr dtrace_io 4 , > .Xr dtrace_ip 4 , > .Xr dtrace_kinst 4 , > diff --git a/share/man/man4/Makefile b/share/man/man4/Makefile > index 48fead7c6287..f5eca038e800 100644 > --- a/share/man/man4/Makefile > +++ b/share/man/man4/Makefile > @@ -940,6 +940,7 @@ _ccd.4= ccd.4 > > .if ${MK_CDDL} != "no" > _dtrace_provs= dtrace_audit.4 \ > + dtrace_fbt.4 \ > dtrace_io.4 \ > dtrace_ip.4 \ > dtrace_kinst.4 \ > diff --git a/share/man/man4/dtrace_fbt.4 b/share/man/man4/dtrace_fbt.4 > new file mode 100644 > index 000000000000..3e35bb8c5bbc > --- /dev/null > +++ b/share/man/man4/dtrace_fbt.4 > @@ -0,0 +1,332 @@ > +.\" > +.\" SPDX-License-Identifier: BSD-2-Clause > +.\" > +.\" Copyright (c) 2025 Mateusz Piotrowski <0mp@FreeBSD.org> > +.\" > +.Dd July 16, 2025 > +.Dt DTRACE_FBT 4 > +.Os > +.Sh NAME > +.Nm dtrace_fbt > +.Nd a DTrace provider for dynamic kernel tracing based on function boundaries > +.Sh SYNOPSIS > +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry > +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return > +.Sh DESCRIPTION > +The Function Boundary Tracing > +.Pq Nm fbt > +provider instruments the entry and return of almost every kernel function > +corresponding to an > +.Xr elf 5 > +symbol in the kernel and loaded kernel modules. > +.Pp > +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry > +fires whenever the > +.Ar function > +is called. > +.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return > +fires when the > +.Ar function > +returns. > +.Pp > +The > +.Ar module > +in the probe description is either the name of the loaded kernel module > +or > +.Ql kernel > +for functions compiled into the kernel. > +.Ss Function Boundary Instrumentation > +The > +.Nm fbt > +will always instrument a function's entry, but > +its return will be intsrumented so long as it can find a > +.Ql ret > +instruction. > +.Pp > +In some cases, > +.Nm fbt > +cannot instrument a function's entry and/or return. > +Refer to subsection > +.Sx Frame Pointer > +for more details. > +.Ss Probe Arguments > +The arguments of the entry probe > +.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry > +are the arguments of the traced function call. > +.Bl -column -offset indent "Entry Probe Argument" "Definition" > +.It Sy Entry Probe Argument Ta Sy Definition > +.It Fa args[0] Ta Function's first argument, typed > +.Pq e.g., Xr malloc 9 Ap s Ft size_t Fa size > +.It Fa args[1] Ta Function's second argument, typed > +.Pq e.g., Xr malloc 9 Ap s Ft struct malloc_type Fa *type > +.It Fa args[2] Ta Function's third argument, typed > +.Pq e.g., Xr malloc 9 Ap s Ft int Fa flags > +.It Fa ... Ta ... > +.El > +.Pp > +The arguments of the return probe > +.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return > +are > +.Fa args[0] > +.Po > +the offset of the firing return instruction within the function; > +useful to tell apart two different return statements in a single function > +.Pc > +and > +.Fa args[1] > +.Pq the return value, if any . > +.Bl -column -offset indent "Return Probe Argument" "Definition" > +.It Sy Return Probe Argument Ta Sy Definition > +.It Fa args[0] Ta Offset of the traced return instruction > +.It Fa args[1] Ta Function's return value > +.Po e.g., a kernel virtual address if returning from a successful > +.Xr malloc 9 > +.Pc > +.El > +.Pp > +Subsection > +.Sx Example 2 : Getting Details About Probe's Arguments > +shows how to get probe's argument count and types directly with > +.Xr dtrace 1 > +without having to resort to the reading function's source code > +or documentation. > +.Sh EXAMPLES > +.Ss Example 1 : Listing Available FBT Probes > +The following example shows how to list all the available > +.Nm fbt > +probes. > +.Bd -literal -offset 2n > +# dtrace -l -P fbt > + ID PROVIDER MODULE FUNCTION NAME > +[...] > +31868 fbt kernel hammer_time entry > +31869 fbt kernel hammer_time return > +[...] > +.Ed > +.Pp > +Since > +.Fn hammer_time > +is a part of the kernel and not a separate loaded module, the > +.Ar module > +column displays > +.Ql kernel . > +.Ss Example 2 : Getting Details About Probe's Arguments > +The following example shows how to generate a program stability report of > +.Xr malloc 9 Ap s > +entry and return probes. > +Those reports are useful to view > +the probe's number of arguments and their types. > +.Bd -literal -offset 2n > +# dtrace -l -v -n fbt::malloc:entry > +[...] > + Argument Types > + args[0]: size_t > + args[1]: struct malloc_type * > + args[2]: int > +.Ed > +.Pp > +The count and types of > +.Nm fbt Ns Cm \&::malloc:entry > +arguments > +match the function signature of > +.Xr malloc 9 : > +.Va args[0] > +is > +.Ft size_t , > +.Va args[1] > +is > +.Ft "struct malloc_type *" , > +and > +.Va "args[2]" > +is > +.Ft int . > +.Bd -literal -offset 2n > +# dtrace -l -v -n fbt::malloc:return > +[...] > + Argument Types > + args[0]: int > + args[1]: void * > +.Ed > +.Pp > +The > +.Cm return > +probe reports two arguments and their types: > +the return instruction offset > +.Pq the usual Ft int > +and the function's return value, which in this case is > +.Ft void * , > +as > +.Xr malloc 9 > +returns a kernel virtual address. > +.Ss Example 3 : Counting Kernel Slab Memory Allocation by Function > +.Bd -literal -offset 2n > +# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }' > +dtrace: description 'fbt::kmem*:entry ' matched 47 probes > +^C > + kmem_alloc_contig 1 > + kmem_alloc_contig_domainset 1 > + kmem_cache_reap_active 1 > + kmem_alloc_contig_pages 2 > + kmem_free 2 > + kmem_std_destructor 19 > + kmem_std_constructor 26 > + kmem_cache_free 151 > + kmem_cache_alloc 181 > +.Ed > +.Ss Example 4 : Counting Kernel Slab Memory Allocation by Calling Function > +.Bd -literal -offset 2n > +# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\en", @); }' > +^C > + kernel`contigmalloc+0x33 1 > + kernel`free+0xd3 1 > + kernel`kmem_alloc_contig+0x29 1 > +kernel`kmem_alloc_contig_domainset+0x19a 1 > + zfs.ko`arc_reap_cb_check+0x16 1 > +.Ed > +.Ss Example 5 : Counting Kernel malloc()'s by Calling Function > +.Bd -literal -offset 2n > +# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\en", @); }' > +^C > + kernel`devclass_get_devices+0xa8 1 > + kernel`sys_ioctl+0xb7 1 > + dtrace.ko`dtrace_ioctl+0x15c1 1 > + dtrace.ko`dtrace_ioctl+0x972 2 > + dtrace.ko`dtrace_dof_create+0x35 2 > + kernel`kern_poll_kfds+0x2f0 4 > + kernel`kern_poll_kfds+0x28a 19 > +.Ed > +.Ss Example 6 : Counting Kernel malloc()'s by Kernel Stack Trace > +.Bd -literal -offset 2n > +# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }' > +^C > + dtrace.ko`dtrace_dof_create+0x35 > + dtrace.ko`dtrace_ioctl+0x827 > + kernel`devfs_ioctl+0xd1 > + kernel`VOP_IOCTL_APV+0x2a > + kernel`vn_ioctl+0xb6 > + kernel`devfs_ioctl_f+0x1e > + kernel`kern_ioctl+0x286 > + kernel`sys_ioctl+0x12f > + kernel`amd64_syscall+0x169 > + kernel`0xffffffff81092b0b > + 2 > +.Ed > +.Ss Example 7 : Summarizing vmem_alloc()'s by Arena Name and Size Distribution > +.Bd -literal -offset 2n > +# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }' > +^C > + > + kernel arena dom > + value ------------- Distribution ------------- count > + 2048 | 0 > + 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4 > + 8192 |@@@@@@@@@@@@@ 2 > + 16384 | 0 > +.Ed > +.Ss Example 8 : Measuring Total Time Spent Executing a Function > +This DTrace script measures the total time spent in > +.Fn vm_page* > +kernel functions. > +The > +.Fn quantize > +aggregation organizes the measurements into power-of-two buckets, > +providing a time distribution in nanoseconds for each function. > +.Bd -literal -offset 2n > +fbt::vm_page*:entry { > + self->start = timestamp; > +} > + > +fbt::vm_page*:return /self->start/ { > + @[probefunc] = quantize(timestamp - self->start); > + self->start = 0; > +} > +.Ed > +.Sh SEE ALSO > +.Xr dtrace 1 , > +.Xr dtrace_kinst 4 , > +.Xr tracing 7 > +.Rs > +.%A Brendan Gregg > +.%A Jim Mauro > +.%B DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD > +.%I Prentice Hall > +.%P pp. 898\(en903 > +.%D 2011 > +.%U https://www.brendangregg.com/dtracebook/ > +.Re > +.Rs > +.%B The illumos Dynamic Tracing Guide > +.%O Chapter fbt Provider > +.%D 2008 > +.%U https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt > +.Re > +.Sh AUTHORS > +This manual page was written by > +.An Mateusz Piotrowski Aq Mt 0mp@FreeBSD.org . > +.Sh CAVEATS > +.Ss Stability and Portability > +.Nm fbt > +probes are by definition tightly coupled to kernel code; if the code underlying > +a script changes, the script may fail to run or may produce incorrect results. > +Scripts written for one version of > +.Fx > +might not work on others, > +and almost certainly will not work on other operating systems. > +.Pp > +Individual > +.Nm fbt > +probes often do not correspond nicely to logical system events. > +For example, consider a DTrace script which prints the destination > +address of every IP packet as the kernel hands them over > +to the network card driver (NIC). > +An > +.Nm fbt Ns -based > +implementation of such a script is a discouragingly difficult task: > +it involves instrumenting at least four different functions in different parts > +of the IPv4 and IPv6 code. > +At the same time, with the > +.Xr dtrace_ip 4 > +provider the script is a simple one-liner: > +.Dl dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}' > +.Pp > +Make sure to review available > +.Xr dtrace 1 > +providers first > +before implementing a custom script with the > +.Nm fbt > +provider. > +If none of the DTrace providers offer the desired probes, > +consider adding new statically-defined tracing probes > +.Pq Xr SDT 9 . > +.Ss Frame Pointer > +Inline functions are not instrumentable by > +.Nm fbt > +as they lack a frame pointer. > +A developer might explicitly disable inlining by adding the > +.Ql __noinline > +attribute to a function definition, > +but of course this requires a recompilation of the kernel. > +Building the kernel with > +.Fl fno-omit-frame-pointer > +is another way of preserving frame pointers. > +Note, that sometimes compilers will omit the frame pointer in leaf functions, > +even when configured with > +.Fl fno-omit-frame-pointer . > +.Pp > +Function returns via a tail call are also not instrumentable by > +.Nm fbt . > +As a result, > +a function might have an entry probe > +and a mix of instrumented and uninstrumentable returns. > +.Pp > +Use > +.Xr dtrace_kinst 4 > +to trace arbitrary instructions inside kernel functions > +and work around some of the > +limitations > +of > +.Nm fbt . > +.Ss Tracing DTrace > +The > +.Nm fbt > +provider cannot attach to functions inside DTrace provider kernel modules. > diff --git a/share/man/man4/dtrace_kinst.4 b/share/man/man4/dtrace_kinst.4 > index 9debbc1bd106..c2187689749b 100644 > --- a/share/man/man4/dtrace_kinst.4 > +++ b/share/man/man4/dtrace_kinst.4 > @@ -22,7 +22,7 @@ > .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > .\" SUCH DAMAGE. > .\" > -.Dd February 27, 2023 > +.Dd July 16, 2025 > .Dt DTRACE_KINST 4 > .Os > .Sh NAME > @@ -43,10 +43,13 @@ creates probes on-demand, meaning it searches for and parses the function's > instructions each time > .Xr dtrace 1 > is run, and not at module load time. > -This is in contrast to FBT's load-time parsing, since > +This is in contrast to > +.Xr dtrace_fbt 4 Ap s > +load-time parsing, since > .Nm kinst > can potentially create thousands of probes for just a single function, instead > -of up to two (entry and return) in the case of FBT. > +of up to two (entry and return) in the case of > +.Xr dtrace_fbt 4 . > A result of this is that > .Cm dtrace -l -P kinst > will not match any probes. > @@ -79,7 +82,8 @@ Trace all instructions in > # dtrace -n 'kinst::amd64_syscall:' > .Ed > .Sh SEE ALSO > -.Xr dtrace 1 > +.Xr dtrace 1 , > +.Xr dtrace_fbt 4 > .Sh HISTORY > The > .Nm kinst > -- Bjoern A. Zeeb r15:7
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?sroo78rp-5r3r-1qo3-84on-r316rq9qo582>