From nobody Mon Mar 6 09:11:41 2023 X-Original-To: freebsd-dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PVXrh2qmDz3wfW5 for ; Mon, 6 Mar 2023 09:11:48 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PVXrf5TvBz47rt; Mon, 6 Mar 2023 09:11:46 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=margiolis.net header.s=default header.b=T3ac06ud; spf=softfail (mx1.freebsd.org: 95.179.159.8 is neither permitted nor denied by domain of christos@freebsd.org) smtp.mailfrom=christos@freebsd.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=default; bh=pv4ljqpj1tEy X7OP59ZUSidThXWdkgKLRB7Epfc9KT4=; h=subject:cc:to:from:date; d=margiolis.net; b=T3ac06udDQj5fvCNqiTQ4FZKYhwzSIodpwA6hNDyXQzVkwAtPOP QjYSouUXOp6gtFfY9FhTh77ISLiAz6hLpJWIvcIkDM64uPekfH4ILfWq/7cdUtHFoW/LAz +Ieg0WmLnDtyzgdXLdBvNwLFlyLUw+SkwbT+XjEt06DVEsgYEc= Received: from pleb (ppp-94-66-59-44.home.otenet.gr [94.66.59.44]) by margiolis.net (OpenSMTPD) with ESMTPSA id 30bb097e (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Mon, 6 Mar 2023 09:11:43 +0000 (UTC) Date: Mon, 6 Mar 2023 11:11:41 +0200 From: Christos Margiolis To: status-updates@freebsdfoundation.org Cc: freebsd-dtrace@freebsd.org, markj@freebsd.org, jrm@freebsd.org Subject: [Development report #4] Improve the kinst DTrace provider Message-ID: <20230306091141.ybjzlz3h4tjjrx64@pleb> List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Result: default: False [-2.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[margiolis.net:s=default]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-dtrace@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[margiolis.net:+]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FREEFALL_USER(0.00)[christos]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4PVXrf5TvBz47rt X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N The past week I submitted a PR for the inline function tracing code I've been working on [1], which fixes the `return` probe bug I mentioned in my last report. The approach for implementing return probes is roughly the following: We figure out whether the DIE of the inline copy has the DW_AT_low_pc and DW_AT_high_pc attributes, in which case the return address is [DW_AT_lowpc + DW_AT_highpc], or if it has the DW_AT_ranges attribute set, in which case we make return probes for the higher boundary of each range. But that's not exactly true because DWARF considers the return address to be that *after* the last instruction: $ ./inlinecall cam_strvis_flag /usr/lib/debug/boot/kernel/kernel.debug /usr/src/sys/cam/cam.c:131 [0xffffffff80353119 - 0xffffffff80353173] /usr/src/sys/cam/cam.c:126 cam_strvis() [0xffffffff8035317e - 0xffffffff803531e1] /usr/src/sys/cam/cam.c:126 cam_strvis() If we look at what instructions 0xffffffff80353173 and 0xffffffff803531e1 correspond: (kgdb) disas /r cam_strvis ... 0xffffffff8035316e <+110>: call 0xffffffff80a8e2c0 0xffffffff80353173 <+115>: add $0x48,%rsp ... 0xffffffff803531df <+223>: jmp 0xffffffff8035319b 0xffffffff803531e1 <+225>: data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 0xffffffff803531e1 in fact is outside cam_strvis() altogether. So to get around this I use LibELF and dtrace_disx86() to go back one instruction and put a probe there. The return offset then is calculated as `inline_copy_retaddr - caller_func_entryaddr`, so for cam_strvis_flag(), we would end up with the offsets 110 and 223: # dtrace -dn 'kinst::cam_strvis_flag:return' dtrace:::ERROR { ((self->%error) = 0x1); } kinst::cam_strvis:110 { } kinst::cam_strvis:223 { } dtrace: description 'kinst::cam_strvis_flag:return' matched 3 probes ... So far this seems to be working fine, but I'll need to do more extensive testing, as well as look into how gdb handles inline frames, to make sure this mechanism is robust enough. Christos [1] https://reviews.freebsd.org/D38825 From nobody Sat Mar 18 20:37:36 2023 X-Original-To: freebsd-dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PfCVY4Mhfz400js for ; Sat, 18 Mar 2023 20:37:41 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PfCVX2czQz3syy; Sat, 18 Mar 2023 20:37:40 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=margiolis.net header.s=default header.b=n1u5Ak8T; spf=softfail (mx1.freebsd.org: 95.179.159.8 is neither permitted nor denied by domain of christos@freebsd.org) smtp.mailfrom=christos@freebsd.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=default; bh=P8B7FpVtaV0N XrmnYyHpp6W3aGBB94BDueGGZgdG2LU=; h=subject:cc:to:from:date; d=margiolis.net; b=n1u5Ak8TQ27QRQgKIGHryhzTIWhexQW5iRYH4B2NmCATuKCpzS9 L9iMMN9Lnet22o504wEuQAiP67L7qmxM81xzKPd7YwZsU1zZpOiyg1PlQ9WVboEMfGJAKD qqyX/2IElZ2A+12OOeomSLlHLjYKVzvR+1EkjdPAUNLUlczbXw= Received: from pleb (ppp-94-66-59-44.home.otenet.gr [94.66.59.44]) by margiolis.net (OpenSMTPD) with ESMTPSA id 53024766 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Sat, 18 Mar 2023 20:37:37 +0000 (UTC) Date: Sat, 18 Mar 2023 22:37:36 +0200 From: Christos Margiolis To: status-updates@freebsdfoundation.org Cc: freebsd-dtrace@freebsd.org, markj@freebsd.org, jrm@freebsd.org Subject: [Development report #5] Improve the kinst DTrace provider Message-ID: <20230318203736.twq3mvsvneqc7hjn@pleb> List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Result: default: False [-2.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[margiolis.net:s=default]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-dtrace@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[margiolis.net:+]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FREEFALL_USER(0.00)[christos]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4PfCVX2czQz3syy X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N I changed the implementation of inline function tracing in dt_sugar to append new probes to the main clause list rather than create separate probes for each one of the inline copies found. Before: # dtrace -dn 'kinst::cam_iosched_has_more_trim:entry' dtrace:::ERROR { ((self->%error) = 0x1); } kinst::cam_iosched_get_trim:13 { } kinst::cam_iosched_next_bio:13 { } kinst::cam_iosched_schedule:40 { } After: # dtrace -dn 'kinst::cam_iosched_has_more_trim:entry' kinst::cam_iosched_get_trim:13, kinst::cam_iosched_next_bio:13, kinst::cam_iosched_schedule:40 { } This turned out to be way simpler and cleaner than the original, broken, mechanism, which was to get a deep copy of the main clause list and copy its predicates and actions to the new clauses. In my last report I mentioned that for return probes, the return offset given by DWARF corresponds to the instruction after the return one, and so I manually went one instruction back to get the correct offset. I realized that doing this unconditionally didn't exactly work as expected, but not doing this didn't work either, because sometimes we end up outside the inline copy's and/or the caller function's bounds. To fix this, we go back one instruction if one of the following conditions is met: - The address returned by DWARF corresponds to the last instruction of the caller function, which in this case (see previous report) is outside the caller function altogether. - If the inline low/high boundaries are given by DW_AT_ranges, we go back one instruction only at the last range. - If the boundaries are given by DW_AT_lowpc/DW_AT_highpc, we always go back one instruction, otherwise we'll end up outside the inline function's bounds. kinst up until now was ignoring any function whose first instruction wasn't `push %rbp`, but doing this means we couldn't trace functions that just happen to `push %rbp` after the first instruction. I modified kinst to instead look if there's a `push %rbp` anywhere in the function. This however still isn't quite complete because kinst did and still does ignore functions that do not `push %rbp` at all, which in some cases is correct, because such functions usually correspond to exception handlers, but there are functions such as cpu_switch(), or leaf functions with their frame pointer omitted (if compiled without `-mno-omit-leaf-frame-pointer`) which are safe to trace, but kinst cannot. Christos