Date: Tue, 28 Jul 2026 23:35:09 +0000 From: Devin Teske <dteske@FreeBSD.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: git: 434283fda99e - main - dwatch: add nine diagnostic modules; grow errno, io, proc, sched Message-ID: <6a693cad.3708f.6ece9ba5@gitrepo.freebsd.org>
index | next in thread | raw e-mail
The branch main has been updated by dteske: URL: https://cgit.FreeBSD.org/src/commit/?id=434283fda99e89af20c3fe95fde427624ae96d04 commit 434283fda99e89af20c3fe95fde427624ae96d04 Author: Devin Teske <dteske@FreeBSD.org> AuthorDate: 2026-07-28 23:32:57 +0000 Commit: Devin Teske <dteske@FreeBSD.org> CommitDate: 2026-07-28 23:32:57 +0000 dwatch: add nine diagnostic modules; grow errno, io, proc, sched Grow the module collection to answer, each with a single command, the first questions asked when diagnosing a sick system: why is my application stalling, where is the kernel fighting over locks, what file could it not find, why is this process getting EPERM, what killed my process, will that fatal signal actually leave a core behind, what was my process stuck on, where is my kernel memory going, who is creating or entering jails, is the network slow because TCP is resending, how long did my thread wait to run, and is the disk itself slow. Every module keeps to the house style: invocation-name overloading through hard links, predicate-only D with inline lookup tables (no if-statements), and stable providers only (syscall, proc, sched, io, dtmalloc, and the lockstat, vfs, priv, and mib SDT providers), so the modules remain drop-in compatible with older releases (the one documented exception is noted below). No kernel changes: new and extended profiles under cddl/usr.sbin/dwatch/libexec plus one libdtrace inline table (priv.d). slow (slow-fsync, slow-open, slow-read, slow-syscall, slow-write, or any slow-NAME by new link) records syscall entry timestamps in thread-local storage and prints, at return, any call whose latency meets a threshold (DWATCH_SLOW_MS, default 100), naming the syscall, the elapsed time to the microsecond, and any errno returned. The bare profile watches a curated set of filesystem-related calls expected to be fast; slow-syscall watches everything; unrecognized invocation names fall through to syscall::NAME:return with the matching entry probe derived mechanically from the return probe list. lock (lock-adaptive, lock-block, lock-lockmgr, lock-rw, lock-spin, lock-sx, lock-thread) rides the dtrace_lockstat(4) block and spin probes, printing the held-off thread (free from the standard event tag), the holdoff duration, the lock class, the lo_name of the lock through a single cast of arg0 to struct lock_object (the first member of every kernel lock), and reader/writer intent on the probes that report it. Holdoffs shorter than DWATCH_LOCK_MS (default 1; 0 shows everything) are suppressed. namei (namei-enoent, namei-entry, namei-failure) records the pathname at vfs:namei:lookup:entry and reports it with the result at return. Unlike the vop_lookup profile, which reconstructs paths from the name cache one component at a time, this sees the whole path exactly as the process requested it. namei-enoent hunts file-not-found storms -- the single most common use of truss(1) -- without stopping the victim. priv (priv-err, priv-ok) watches priv_check(9) verdicts, naming the exact privilege denied -- something no syscall tracer can see, because by the time EPERM surfaces the priv(9) value is gone. The number is decoded by priv_string[], a new libdtrace inline table in the errno.d and signal.d tradition, mechanically generated from sys/priv.h (247 entries) and installed to /usr/lib/dtrace where dtrace(1) auto-loads it; on older releases it is a drop-in file like the module itself. coredump (coredump-top) watches for delivery of signals whose default action produces a core, per the SIGPROP_CORE entries of the sigproptbl in kern_sig.c, and renders a verdict the same way and in the same order the kernel will decide it: ignored or caught per the target's struct sigacts, then the coredump() gauntlet of kern.coredump, kern.sugid_coredump vs P_SUGID, procctl(2) PROC_TRACE_CTL, and RLIMIT_CORE -- the sysctl knobs read live through kernel globals. Where a coredump-worthy signal will produce no core, the verdict says precisely which policy ate it. coredump-top maintains a cumulative catalog of coredump-worthy signals by process and signal, refreshed every 3 seconds in the style of systop; combine the event profile with `-O cmd' to capture state as each event occurs. hang (hang-top) pairs sched:::sleep with sched:::wakeup through a tid-keyed timestamp array and prints, as each thread wakes, any sleep that meets a threshold (DWATCH_HANG_MS, default 1000), naming the sleeper in the details and the waker in the standard event tag. This is the blocking the slow module structurally cannot see: a syscall that never returns never reports its latency, while hang reports the moment the wait ends, with the full duration. hang-top maintains a cumulative catalog of long sleeps by process (count and maximum) in the style of coredump-top. jail (jail-attach, jail-get, jail-remove, jail-set) watches the jail management plane -- jail(2), jail_set(2), jail_get(2), jail_attach(2), and jail_remove(2) -- naming the operation, the jail id (taken from the entry argument for attach/remove, from the return value for the others), and any errno. Complements the dwatch `-j jail' filter, which scopes any profile to processes inside one jail; this watches who manipulates jails, from any jail or none. dtmalloc (dtmalloc-top, or any dtmalloc-NAME by new link) rides the dtmalloc provider (one malloc and one free probe per malloc(9) type). The event profile prints allocations and frees meeting a size threshold (DWATCH_MALLOC_MIN, default 65536) -- who is allocating huge kernel buffers. dtmalloc-top maintains a running catalog of net bytes and outstanding allocation balance by type, sorted by net bytes so leak suspects rise: a type that climbs without bound while the system is in steady state is the suspect. The catalog reflects activity since the watch began, and is honest about caches holding what they allocate. mib (tcp-retransmit, or any mib-NAME by new link) rides the per-counter mib SDT probes of the network stack. The tcp-retransmit profile curates the counters that signal send-path congestion or loss -- data packet retransmissions, unnecessary retransmissions, retransmit timer expirations, and connections dropped by retransmit exhaustion -- decoded through an inline description table, answering "is this network slow because TCP is resending?" as events with process context rather than netstat(1) deltas. NB: the mib probes exist only in kernels built with options KDTRACE_MIB_SDT (default in -CURRENT via std.debug); the module documents this and dtrace(1) refuses the script elsewhere, making the dependency self-announcing. Four existing modules gain personalities. proc grows proc-signal-fatal, filtering signal-send to signals whose default disposition terminates the receiver, most-notably including kernel-generated SIGSEGV/SIGBUS/SIGILL/SIGFPE that no kill(2) watcher will ever see. errno now reads its invocation name: errno-NAME shows only syscalls returning that errno, where NAME is a symbolic name from errno.d or a number; links are installed for errno-EACCES, errno-ECAPMODE, errno-ENOENT, errno-ENOTCAPABLE, and errno-EPERM (the latter pairs covering capsicum(4) capability-mode violations), and any other errno needs only a new link. sched grows sched-latency, recording a timestamp at sched:::enqueue keyed by tid and printing at sched:::on-cpu any run-queue wait meeting a threshold (DWATCH_SCHED_MS, default 10) -- the literal measurement of scheduler delay on a system with idle CPU that still feels sluggish. io grows io-slow, pairing io:::start with io:::done through a bio-keyed timestamp array and printing any request that meets a threshold (DWATCH_IO_MS, default 100), naming the device, command, size, and elapsed time; watched against zvols and a pool's leaf vdevs this brackets where in a ZFS stack the time is going, without touching unstable providers. Document all of the above plus the DWATCH_HANG_MS, DWATCH_IO_MS, DWATCH_LOCK_MS, DWATCH_MALLOC_MIN, DWATCH_SCHED_MS, and DWATCH_SLOW_MS knobs in dwatch(1). All 46 new invocation names were exercised through `dwatch -d' with a profile-path sandbox emulating the installed hard links: every one sources cleanly and emits the intended D -- probe selection per alias, entry/return and sleep/wakeup pairing through thread-local and global associative arrays, threshold and mask predicates picking up their knobs, aggregation clauses and printa column layout in the -top profiles, multi-line predicate rendering, and `-t' correctly displacing each module's default test were verified by inspection of the generated scripts. Invocations untouched by this pass generate D identical to their previous output. Modules pass sh -n, fit 80 columns, and dwatch.1 passes mandoc -Tlint with no new warnings. A validation harness performing a `dwatch -e' compile per profile against the live kernel globs every staged profile for runs wherever the dtrace device is present. Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D58093 --- cddl/lib/libdtrace/Makefile | 6 +- cddl/lib/libdtrace/mkpriv | 119 +++++++++++++++++++ cddl/usr.sbin/dwatch/dwatch.1 | 105 ++++++++++++++++- cddl/usr.sbin/dwatch/libexec/Makefile | 42 +++++++ cddl/usr.sbin/dwatch/libexec/coredump | 210 ++++++++++++++++++++++++++++++++++ cddl/usr.sbin/dwatch/libexec/dtmalloc | 146 +++++++++++++++++++++++ cddl/usr.sbin/dwatch/libexec/errno | 15 ++- cddl/usr.sbin/dwatch/libexec/hang | 174 ++++++++++++++++++++++++++++ cddl/usr.sbin/dwatch/libexec/io | 72 +++++++++++- cddl/usr.sbin/dwatch/libexec/jail | 92 +++++++++++++++ cddl/usr.sbin/dwatch/libexec/lock | 136 ++++++++++++++++++++++ cddl/usr.sbin/dwatch/libexec/mib | 78 +++++++++++++ cddl/usr.sbin/dwatch/libexec/namei | 84 ++++++++++++++ cddl/usr.sbin/dwatch/libexec/priv | 59 ++++++++++ cddl/usr.sbin/dwatch/libexec/proc | 23 +++- cddl/usr.sbin/dwatch/libexec/sched | 65 ++++++++++- cddl/usr.sbin/dwatch/libexec/slow | 131 +++++++++++++++++++++ 17 files changed, 1544 insertions(+), 13 deletions(-) diff --git a/cddl/lib/libdtrace/Makefile b/cddl/lib/libdtrace/Makefile index fbc6d5fbbec2..08836a91b164 100644 --- a/cddl/lib/libdtrace/Makefile +++ b/cddl/lib/libdtrace/Makefile @@ -55,6 +55,7 @@ DSRCS= cam.d \ errno.d \ io.d \ ip.d \ + priv.d \ psinfo.d \ sctp.d \ siftr.d \ @@ -150,7 +151,7 @@ LDFLAGS+= -fsanitize=address -fsanitize=undefined LIBADD= ctf elf proc pthread rtld_db xo -CLEANFILES= dt_errtags.c dt_names.c +CLEANFILES= dt_errtags.c dt_names.c priv.d .include <bsd.lib.mk> @@ -159,3 +160,6 @@ dt_errtags.c: ${OPENSOLARIS_USR_DISTDIR}/lib/libdtrace/common/dt_errtags.h dt_names.c: ${OPENSOLARIS_SYS_DISTDIR}/uts/common/sys/dtrace.h sh ${OPENSOLARIS_USR_DISTDIR}/lib/libdtrace/common/mknames.sh < ${.ALLSRC} > ${.TARGET} + +priv.d: mkpriv ${SRCTOP}/sys/sys/priv.h + sh ${.CURDIR}/mkpriv -o ${.TARGET} ${SRCTOP}/sys/sys/priv.h diff --git a/cddl/lib/libdtrace/mkpriv b/cddl/lib/libdtrace/mkpriv new file mode 100755 index 000000000000..d213161f8356 --- /dev/null +++ b/cddl/lib/libdtrace/mkpriv @@ -0,0 +1,119 @@ +#!/bin/sh +# -*- tab-width: 4 -*- ;; Emacs +# vi: set filetype=sh tabstop=8 shiftwidth=8 noexpandtab :: Vi/ViM +############################################################ LICENSE +# +# Copyright (c) 2026 Devin Teske <dteske@FreeBSD.org> +# +# SPDX-License-Identifier: BSD-2-Clause +# +############################################################ IDENT(1) +# +# $Title: Generate priv.d from <sys/priv.h> for libdtrace(3) $ +# $Copyright: 2026 Devin Teske. All rights reserved. $ +# +############################################################ GLOBALS + +VERSION='$Version: 1.0 $' + +pgm="${0##*/}" # Program basename + +# +# Global exit status +# +SUCCESS=0 +FAILURE=1 + +# +# Command-line options +# +OUTPUT_FILE= # -o file + +# +# Miscellaneous +# +PRIV_H= + +############################################################ FUNCTIONS + +usage() +{ + local fmt="$1" + local optfmt="\t%-5s %s\n" + + exec >&2 + if [ "$fmt" ]; then + shift 1 # fmt + printf "%s: $fmt\n" "$pgm" "$@" + fi + printf "Usage: %s [-hv] [-o file] priv.h\n" "$pgm" + printf "Options:\n" + printf "$optfmt" "-h" "Print this usage statment and exit." + printf "$optfmt" "-o file" "Write to output file instead of stdout." + printf "$optfmt" "-v" "Print version information and exit." + exit $FAILURE +} + +############################################################ MAIN + +# +# Process command-line options +# +while getopts ho:v flag; do + case "$flag" in + o) OUTPUT_FILE="$OPTARG" ;; + v) VERSION="${VERSION#*: }" + echo "${VERSION% $}" + exit $SUCCESS ;; + *) usage # NOTREACHED + esac +done +shift $(( $OPTIND - 1 )) + +# +# Check command-line arguments +# +[ $# -gt 0 ] || usage "Too few arguments" # NOTREACHED +[ $# -eq 1 ] || usage "Too many arguments" # NOTREACHED +[ "$1" ] || usage "Missing priv.h argument" # NOTREACHED +[ -e "$1" ] || usage "%s: No such file or directory" "$1" # NOTREACHED + +# +# Generate priv.d contents from priv.h input +# +PRIV_H="$1" +exec 9<<PRIV_D_EOF +/*- + * Copyright (c) 2026 Devin Teske <dteske@FreeBSD.org> + * + * SPDX-License-Identifier: BSD-2-Clause + */ + +/* + * Map priv(9) privilege values to their symbolic names. + * NB: Mechanically generated from <sys/priv.h> -- keep in-sync. + * NB: Generated by /usr/src/cddl/lib/libdtrace/$pgm + */ + +#pragma D binding "1.13" priv_string +inline string priv_string[int priv] = +$( awk ' + /^#define[[:space:]]+PRIV_[A-Z0-9_]+[[:space:]]+[0-9]+/ { + for (i = 0; ++i <= NF; ) if ($i ~ /define/) break + name = $++i + value = $++i + printf "\tpriv == %s ?\t\t\"%s\" :\n", value, name + }' "$PRIV_H" +) + strjoin("PRIV_UNKNOWN#", lltostr(priv)); +PRIV_D_EOF + +# +# Output contents +# +[ ! "$OUTPUT_FILE" ] || exec > "$OUTPUT_FILE" +cat <&9 + +################################################################################ +# END +################################################################################ diff --git a/cddl/usr.sbin/dwatch/dwatch.1 b/cddl/usr.sbin/dwatch/dwatch.1 index 81efb2c2c437..8c201d970620 100644 --- a/cddl/usr.sbin/dwatch/dwatch.1 +++ b/cddl/usr.sbin/dwatch/dwatch.1 @@ -1,4 +1,4 @@ -.\" Copyright (c) 2014-2018 Devin Teske +.\" Copyright (c) 2014-2026 Devin Teske .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without @@ -22,7 +22,7 @@ .\" ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" -.Dd April 18, 2025 +.Dd July 4, 2026 .Dt DWATCH 1 .Os .Sh NAME @@ -382,17 +382,69 @@ Print mode and path from .Xr chmod 2 , .Xr lchmod 2 , .Xr fchmodat 2 +.It coredump +Print coredump-worthy signal deliveries with a verdict of whether a +core will actually be written +.It coredump-top +Maintain a running catalog of coredump-worthy signals by process and +signal +.It dtmalloc +Print kernel +.Xr malloc 9 +and +.Xr free 9 +activity by malloc type, +meeting a size threshold +.It dtmalloc-top +Maintain a running catalog of net bytes and allocation balance by +malloc type +.Pq leak suspects rise .It errno Print non-zero errno results from system calls +.It errno- Ns Ar NAME +Print system calls returning the given errno, +by symbolic name or number +.Pq e.g., errno-ENOENT, errno-ENOTCAPABLE, errno-13 +.It hang +Print threads that were asleep longer than a threshold, +as they wake, +naming the sleeper and the waker +.It hang-top +Maintain a running catalog of long sleeps by process .It io Print disk I/O details provided by .Xr dtrace_io 4 +.It io-slow +Print disk I/O requests exceeding a latency threshold .It ip Print IPv4 and IPv6 details provided by .Xr dtrace_ip 4 +.It jail +Print jail management activity from +.Xr jail 2 , +.Xr jail_set 2 , +.Xr jail_get 2 , +.Xr jail_attach 2 , +.Xr jail_remove 2 .It kill Print signal and pid from .Xr kill 2 +.It lock +Print kernel lock contention details provided by +.Xr dtrace_lockstat 4 +.It lock-spin +Print kernel lock spinning details provided by +.Xr dtrace_lockstat 4 +.It mib +Print network stack statistics counters as they increment +.Pq requires a kernel built with options KDTRACE_MIB_SDT +.It namei +Print pathname resolution details provided by the +.Fn vfs:namei:lookup +probes +.It namei-failure +Print failed pathname resolutions +.Pq also namei-enoent for ENOENT only .It nanosleep Print requested time from .Xr nanosleep 2 @@ -400,12 +452,19 @@ Print requested time from Print path from .Xr open 2 , .Xr openat 2 +.It priv +Print privilege requests denied by +.Xr priv_check 9 +.Pq also priv-ok for those granted .It proc Print process execution details provided by .Xr dtrace_proc 4 .It proc-signal Print process signal details provided by .Xr dtrace_proc 4 +.It proc-signal-fatal +Print only signals whose default disposition is to terminate the +receiving process .It rw Print buffer contents from .Xr read 2 , @@ -413,12 +472,24 @@ Print buffer contents from .It sched Print CPU scheduling details provided by .Xr dtrace_sched 4 +.It sched-latency +Print threads that waited longer than a threshold between becoming +runnable and running +.It slow +Print filesystem-related syscalls exceeding a latency threshold +.It slow-syscall +Print any syscall exceeding a latency threshold .It tcp Print TCP address/port details provided by .Xr dtrace_tcp 4 .It tcp-io Print TCP I/O details provided by .Xr dtrace_tcp 4 +.It tcp-retransmit +Print TCP retransmissions, +retransmit timeouts, +and connections dropped by retransmit exhaustion +.Pq requires a kernel built with options KDTRACE_MIB_SDT .It udp Print UDP I/O details provided by .Xr dtrace_udp 4 @@ -454,6 +525,26 @@ Print symlink paths being created by These environment variables affect the execution of .Nm : .Bl -tag -width "DWATCH_PROFILES_PATH" +.It Ev DWATCH_HANG_MS +Minimum sleep duration, +in milliseconds, +displayed by the hang profiles +.Pq default 1000; 0 to show everything . +.It Ev DWATCH_IO_MS +Minimum disk I/O request latency, +in milliseconds, +displayed by the io-slow profile +.Pq default 100; 0 to show everything . +.It Ev DWATCH_LOCK_MS +Minimum lock holdoff duration, +in milliseconds, +displayed by the lock profiles +.Pq default 1; 0 to show everything . +.It Ev DWATCH_MALLOC_MIN +Minimum allocation size, +in bytes, +displayed by the dtmalloc profiles +.Pq default 65536; 0 to show everything . .It Ev DWATCH_PROFILES_PATH If .Ev DWATCH_PROFILES_PATH @@ -464,6 +555,16 @@ variable instead of the default .Ql Li /usr/libexec/dwatch:/usr/local/libexec/dwatch . If set to NULL, profiles are not loaded. +.It Ev DWATCH_SCHED_MS +Minimum run-queue wait, +in milliseconds, +displayed by the sched-latency profile +.Pq default 10; 0 to show everything . +.It Ev DWATCH_SLOW_MS +Minimum syscall latency, +in milliseconds, +displayed by the slow profiles +.Pq default 100 . .El .Sh EXIT STATUS .Ex -std diff --git a/cddl/usr.sbin/dwatch/libexec/Makefile b/cddl/usr.sbin/dwatch/libexec/Makefile index 72ea5be686fb..d32c54c34c83 100644 --- a/cddl/usr.sbin/dwatch/libexec/Makefile +++ b/cddl/usr.sbin/dwatch/libexec/Makefile @@ -1,16 +1,25 @@ PACKAGE= dwatch FILESDIR= ${LIBEXECDIR}/dwatch FILES= chmod \ + coredump \ + dtmalloc \ errno \ + hang \ io \ ip \ + jail \ kill \ + lock \ + mib \ + namei \ nanosleep \ open \ + priv \ proc \ rw \ sched \ sendrecv \ + slow \ systop \ tcp \ udp \ @@ -23,11 +32,37 @@ FILES= chmod \ LINKMODE= ${SHAREMODE} LINKS= ${LIBEXECDIR}/dwatch/chmod ${LIBEXECDIR}/dwatch/fchmodat LINKS+= ${LIBEXECDIR}/dwatch/chmod ${LIBEXECDIR}/dwatch/lchmod +LINKS+= ${LIBEXECDIR}/dwatch/coredump ${LIBEXECDIR}/dwatch/coredump-top +LINKS+= ${LIBEXECDIR}/dwatch/dtmalloc ${LIBEXECDIR}/dwatch/dtmalloc-top +LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-EACCES +LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-ECAPMODE +LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-ENOENT +LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-ENOTCAPABLE +LINKS+= ${LIBEXECDIR}/dwatch/errno ${LIBEXECDIR}/dwatch/errno-EPERM +LINKS+= ${LIBEXECDIR}/dwatch/hang ${LIBEXECDIR}/dwatch/hang-top LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-done +LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-slow LINKS+= ${LIBEXECDIR}/dwatch/io ${LIBEXECDIR}/dwatch/io-start LINKS+= ${LIBEXECDIR}/dwatch/ip ${LIBEXECDIR}/dwatch/ip-receive LINKS+= ${LIBEXECDIR}/dwatch/ip ${LIBEXECDIR}/dwatch/ip-send +LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-attach +LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-get +LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-remove +LINKS+= ${LIBEXECDIR}/dwatch/jail ${LIBEXECDIR}/dwatch/jail-set +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-adaptive +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-block +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-lockmgr +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-rw +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-spin +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-sx +LINKS+= ${LIBEXECDIR}/dwatch/lock ${LIBEXECDIR}/dwatch/lock-thread +LINKS+= ${LIBEXECDIR}/dwatch/mib ${LIBEXECDIR}/dwatch/tcp-retransmit +LINKS+= ${LIBEXECDIR}/dwatch/namei ${LIBEXECDIR}/dwatch/namei-enoent +LINKS+= ${LIBEXECDIR}/dwatch/namei ${LIBEXECDIR}/dwatch/namei-entry +LINKS+= ${LIBEXECDIR}/dwatch/namei ${LIBEXECDIR}/dwatch/namei-failure LINKS+= ${LIBEXECDIR}/dwatch/open ${LIBEXECDIR}/dwatch/openat +LINKS+= ${LIBEXECDIR}/dwatch/priv ${LIBEXECDIR}/dwatch/priv-err +LINKS+= ${LIBEXECDIR}/dwatch/priv ${LIBEXECDIR}/dwatch/priv-ok LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-create LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-exec LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-exec-failure @@ -36,6 +71,7 @@ LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-exit LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-clear LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-discard +LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-fatal LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-signal-send LINKS+= ${LIBEXECDIR}/dwatch/proc ${LIBEXECDIR}/dwatch/proc-status LINKS+= ${LIBEXECDIR}/dwatch/rw ${LIBEXECDIR}/dwatch/read @@ -45,6 +81,7 @@ LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-cpu LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-dequeue LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-enqueue LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-exec +LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-latency LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-lend-pri LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-load-change LINKS+= ${LIBEXECDIR}/dwatch/sched ${LIBEXECDIR}/dwatch/sched-off-cpu @@ -63,6 +100,11 @@ LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/recvmsg LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/send LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/sendmsg LINKS+= ${LIBEXECDIR}/dwatch/sendrecv ${LIBEXECDIR}/dwatch/sendto +LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-fsync +LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-open +LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-read +LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-syscall +LINKS+= ${LIBEXECDIR}/dwatch/slow ${LIBEXECDIR}/dwatch/slow-write LINKS+= ${LIBEXECDIR}/dwatch/tcp ${LIBEXECDIR}/dwatch/tcp-accept LINKS+= ${LIBEXECDIR}/dwatch/tcp ${LIBEXECDIR}/dwatch/tcp-accept-established LINKS+= ${LIBEXECDIR}/dwatch/tcp ${LIBEXECDIR}/dwatch/tcp-accept-refused diff --git a/cddl/usr.sbin/dwatch/libexec/coredump b/cddl/usr.sbin/dwatch/libexec/coredump new file mode 100644 index 000000000000..0b68564691de --- /dev/null +++ b/cddl/usr.sbin/dwatch/libexec/coredump @@ -0,0 +1,210 @@ +# -*- tab-width: 4 -*- ;; Emacs +# vi: set filetype=sh tabstop=8 shiftwidth=8 noexpandtab :: Vi/ViM +############################################################ IDENT(1) +# +# $Title: dwatch(8) module for coredump-worthy signal delivery $ +# $Copyright: 2026 Devin Teske. All rights reserved. $ +# +############################################################ DESCRIPTION +# +# Print signals whose default action produces a coredump, as they are +# sent, together with a verdict of whether a core will actually be +# written, rendered the same way and in the same order the kernel will +# decide it: the target may be ignoring or catching the signal, dumps +# may be disabled (kern.coredump), the target may be flagged sugid +# (kern.sugid_coredump) or may have disabled tracing via procctl(2) +# PROC_TRACE_CTL, or RLIMIT_CORE may be 0. +# Coredump-worthy is defined by the SIGPROP_CORE entries of the +# sigproptbl in kern_sig.c: SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGEMT, +# SIGFPE, SIGBUS, SIGSEGV, and SIGSYS. +# The coredump-top profile maintains a running catalog, updated +# every 3 seconds, of coredump-worthy signals by process and signal. +# Combine with `-O cmd' to capture state as each event occurs. +# +############################################################ PRAGMAS + +case "$PROFILE" in +coredump-top) + DTRACE_PRAGMA=" + option quiet + option aggsortrev + " # END-QUOTE + ;; +esac + +############################################################ PROBE + +case "$PROFILE" in +coredump-top) + : ${PROBE:=profile:::tick-3s} ;; +*) + : ${PROBE:=proc:::signal-send} +esac + +############################################################ GLOBALS + +exec 9<<EOF +/* + * Signals whose default action includes a coredump, per the + * SIGPROP_CORE entries of the sigproptbl in kern_sig.c of FreeBSD + */ +inline int coredump_sig[int sig] = + sig == SIGQUIT ? 1 : + sig == SIGILL ? 1 : + sig == SIGTRAP ? 1 : + sig == SIGABRT ? 1 : /* SIGIOT */ + sig == SIGEMT ? 1 : + sig == SIGFPE ? 1 : + sig == SIGBUS ? 1 : + sig == SIGSEGV ? 1 : + sig == SIGSYS ? 1 : + 0; +EOF +GLOBALS=$( cat <&9 ) + +############################################################ EVENT ACTION + +[ "$CUSTOM_TEST" ] || case "$PROFILE" in +coredump-top) ;; +*) EVENT_TEST="coredump_sig[this->sig]" +esac + +############################################################ ACTIONS + +if [ "$PROFILE" = "coredump-top" ]; then +exec 9<<EOF +$GLOBALS + +BEGIN { printf("Cataloging coredump-worthy signals ...") } /* probe ID $ID */ + +proc:::signal-send /coredump_sig[(int)arg2]/ /* probe ID $(( $ID + 1 )) */ +{ + @cores[stringof(((struct proc *)args[1])->p_comm), + signal_string[(int)arg2]] = count(); +} +EOF +ACTIONS=$( cat <&9 ) +ID=$(( $ID + 2 )) +else +exec 9<<EOF +$GLOBALS + +this int sig; +this pid_t pid; +this string verdict; +this struct proc * target; +this u_int core_catch; +this u_int core_ign; +this u_int core_notrace; +this u_int core_sugid; +this uint64_t core_limit; + +$PROBE /* probe ID $ID */ +{${TRACE:+ + printf("<$ID>"); +} + this->target = (struct proc *)args[1]; + this->pid = (pid_t)this->target->p_pid; + this->sig = (int)arg2; +} + +$PROBE /* probe ID $(( $ID + 1 )) */ +{${TRACE:+ + printf("<$(( $ID + 1 ))>"); +} + /* + * Render a verdict the way the kernel will: disposition first + * (struct sigacts), then the coredump() gauntlet in its order: + * kern.coredump, kern.sugid_coredump vs P_SUGID, procctl(2) + * PROC_TRACE_CTL (P2_NOTRACE), and finally RLIMIT_CORE + */ + this->core_ign = + this->target->p_sigacts->ps_sigignore.__bits[ + (this->sig - 1) >> 5] & + (1 << ((this->sig - 1) & 31)); + this->core_catch = + this->target->p_sigacts->ps_sigcatch.__bits[ + (this->sig - 1) >> 5] & + (1 << ((this->sig - 1) & 31)); + this->core_sugid = + this->target->p_flag & 0x00000100; /* P_SUGID */ + this->core_notrace = + this->target->p_flag2 & 0x00000002; /* P2_NOTRACE */ + this->core_limit = this->target->p_limit-> + pl_rlimit[4].rlim_cur; /* RLIMIT_CORE */ + + this->verdict = + this->core_ign ? "ignored" : + this->core_catch ? "caught" : + \`do_coredump == 0 ? "denied by kern.coredump" : + this->core_sugid && \`sugid_coredump == 0 ? + "denied by kern.sugid_coredump" : + this->core_notrace ? "denied by procctl trace ctl" : + this->core_limit == 0 ? "denied by RLIMIT_CORE" : + "will dump core"; + + $( pproc -P _core "(struct proc *)args[1]" ) +} +EOF +ACTIONS=$( cat <&9 ) +ID=$(( $ID + 2 )) +fi + +############################################################ EVENT TAG + +# For the running catalog, override the default `UID.GID CMD[PID]: ' tag +# with ANSI cursor-homing and screen-clearing codes plus column headers. + +if [ "$PROFILE" = "coredump-top" ]; then +size=$( stty size 2> /dev/null ) +rows="${size%% *}" +cols="${size#* }" + +exec 9<<EOF + printf("\033[H"); /* Position the cursor at top-left */ + printf("\033[J"); /* Clear display from cursor to end */ + + /* Header line containing probe (left) and date (right) */ + printf("%-*s%s%Y%s\n", + $(( ${cols:-80} - 20 )), "$PROBE", + console ? "\033[32m" : "", + walltimestamp, + console ? "\033[39m" : ""); + + /* Column headers */ + printf("%s%8s %-20s %s%s\n", + console ? "\033[1m" : "", + "COUNT", + "EXECNAME", + "SIGNAL", + console ? "\033[22m" : ""); +EOF +EVENT_TAG=$( cat <&9 ) +fi + +############################################################ EVENT DETAILS + +if [ "$PROFILE" = "coredump-top" ]; then +exec 9<<EOF + /* NB: Cumulative; not truncated between updates */ + printa("%@8u %-20s %s\n", @cores); +EOF +EVENT_DETAILS=$( cat <&9 ) +elif [ ! "$CUSTOM_DETAILS" ]; then +exec 9<<EOF + /* + * Print coredump-worthy signal details + */ + printf("%s[%d] pid %d (%s) -- %s", + signal_string[this->sig], + this->sig, + this->pid, + this->verdict, + this->args_core); +EOF +EVENT_DETAILS=$( cat <&9 ) +fi + +################################################################################ +# END +################################################################################ diff --git a/cddl/usr.sbin/dwatch/libexec/dtmalloc b/cddl/usr.sbin/dwatch/libexec/dtmalloc new file mode 100644 index 000000000000..ad1a044020fa --- /dev/null +++ b/cddl/usr.sbin/dwatch/libexec/dtmalloc @@ -0,0 +1,146 @@ +# -*- tab-width: 4 -*- ;; Emacs +# vi: set filetype=sh tabstop=8 shiftwidth=8 noexpandtab :: Vi/ViM +############################################################ IDENT(1) +# +# $Title: dwatch(8) module for malloc(9) type activity $ +# $Copyright: 2026 Devin Teske. All rights reserved. $ +# +############################################################ DESCRIPTION +# +# Print kernel malloc(9) and free(9) activity by malloc type, via the +# dtmalloc provider (one malloc and one free probe per type; see +# vmstat -m for the types). The default profile prints allocations and +# frees meeting a size threshold (default 65536 bytes; tunable via +# DWATCH_MALLOC_MIN in the environment, 0 to show everything), +# answering "who is allocating huge kernel buffers?" Use +# dtmalloc-NAME to watch a single type (e.g., dtmalloc-devbuf). +# The dtmalloc-top profile maintains a running catalog, updated every +# 3 seconds, of net bytes and outstanding allocation balance by type: +# a type whose net bytes climb without bound while the system is in +# steady state is a leak suspect. NB: the catalog reflects activity +# since the watch began, not preexisting allocations, and caches +# legitimately hold what they allocate. +# +############################################################ PRAGMAS + +case "$PROFILE" in +dtmalloc-top) + DTRACE_PRAGMA=" + option quiet + option aggsortrev + " # END-QUOTE + ;; +esac + +############################################################ PROBE + +case "$PROFILE" in +dtmalloc) + : ${PROBE:=dtmalloc:::malloc, dtmalloc:::free} ;; +dtmalloc-top) + : ${PROBE:=profile:::tick-3s} ;; +*) + : ${PROBE:=$( echo \ + dtmalloc::${PROFILE#dtmalloc-}:malloc, \ + dtmalloc::${PROFILE#dtmalloc-}:free )} ;; +esac + +############################################################ EVENT ACTION + +: ${DWATCH_MALLOC_MIN:=65536} + +case "$DWATCH_MALLOC_MIN" in +""|*[!0-9]*) die "DWATCH_MALLOC_MIN must be a number" ;; # NOTREACHED +esac + +[ "$CUSTOM_TEST" ] || case "$PROFILE" in +dtmalloc-top) ;; +*) EVENT_TEST="(uint64_t)arg3 >= $DWATCH_MALLOC_MIN" +esac + +############################################################ ACTIONS + +if [ "$PROFILE" = "dtmalloc-top" ]; then +exec 9<<EOF +this int64_t dtmalloc_delta; + +BEGIN { printf("Cataloging malloc(9) activity ...") } /* probe ID $ID */ + +dtmalloc:::malloc /* probe ID $(( $ID + 1 )) */ +{ + this->dtmalloc_delta = (int64_t)arg3; +} + +dtmalloc:::free /* probe ID $(( $ID + 2 )) */ +{ + this->dtmalloc_delta = -(int64_t)arg3; +} + +dtmalloc:::malloc, dtmalloc:::free /* probe ID $(( $ID + 3 )) */ +{ + @dtmalloc_bytes[probefunc] = sum(this->dtmalloc_delta); + @dtmalloc_allocs[probefunc] = + sum(this->dtmalloc_delta > 0 ? 1 : -1); +} +EOF +ACTIONS=$( cat <&9 ) +ID=$(( $ID + 4 )) +fi + +############################################################ EVENT TAG + +# For the running catalog, override the default `UID.GID CMD[PID]: ' tag +# with ANSI cursor-homing and screen-clearing codes plus column headers. + +if [ "$PROFILE" = "dtmalloc-top" ]; then +size=$( stty size 2> /dev/null ) +rows="${size%% *}" +cols="${size#* }" + +exec 9<<EOF + printf("\033[H"); /* Position the cursor at top-left */ + printf("\033[J"); /* Clear display from cursor to end */ + + /* Header line containing probe (left) and date (right) */ + printf("%-*s%s%Y%s\n", + $(( ${cols:-80} - 20 )), "$PROBE", + console ? "\033[32m" : "", + walltimestamp, + console ? "\033[39m" : ""); + + /* Column headers */ + printf("%s%14s %10s %s%s\n", + console ? "\033[1m" : "", + "NET(B)", + "BALANCE", + "TYPE", + console ? "\033[22m" : ""); +EOF +EVENT_TAG=$( cat <&9 ) +fi + +############################################################ EVENT DETAILS + +if [ "$PROFILE" = "dtmalloc-top" ]; then +exec 9<<EOF + /* NB: Cumulative; not truncated between updates */ + printa("%@14d %@10d %s\n", @dtmalloc_bytes, @dtmalloc_allocs); +EOF +EVENT_DETAILS=$( cat <&9 ) +elif [ ! "$CUSTOM_DETAILS" ]; then +exec 9<<EOF + /* + * Print malloc(9) activity details + */ + printf("%s(9) type %s %d byte%s", + probename, + probefunc, + (long)arg3, + (long)arg3 == 1 ? "" : "s"); +EOF +EVENT_DETAILS=$( cat <&9 ) +fi + +################################################################################ +# END +################################################################################ diff --git a/cddl/usr.sbin/dwatch/libexec/errno b/cddl/usr.sbin/dwatch/libexec/errno index d899906a1313..e87cbde7ce50 100644 --- a/cddl/usr.sbin/dwatch/libexec/errno +++ b/cddl/usr.sbin/dwatch/libexec/errno @@ -3,7 +3,7 @@ ############################################################ IDENT(1) # # $Title: dwatch(8) module for syscall errno logging $ -# $Copyright: 2014-2018 Devin Teske. All rights reserved. $ +# $Copyright: 2014-2026 Devin Teske. All rights reserved. $ # ############################################################ DESCRIPTION # @@ -11,13 +11,24 @@ *** 1079 LINES SKIPPED ***home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6a693cad.3708f.6ece9ba5>