Date: Thu, 2 Jun 2022 16:55:10 -0400 From: Mark Johnston <markj@freebsd.org> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Dtrace, Rust, and LLVM13 Message-ID: <Ypkjrmmm%2BK34vR00@nuc> In-Reply-To: <CAOtMX2gtpUkYVU1DJ8yswwRtuig2pVB_zSQV6BjszNvN7JA_hQ@mail.gmail.com> References: <CAOtMX2gtpUkYVU1DJ8yswwRtuig2pVB_zSQV6BjszNvN7JA_hQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jun 01, 2022 at 09:15:28PM -0600, Alan Somers wrote: > The best way to profile a Rust program on FreeBSD is with Brendan > Gregg's Flamegraph[1]. This is based on dtrace's ustack() function. > It used to work great. However, Rust v1.56.0 is based on LLVM13[2] > and now dtrace can't print user stacks anymore. For example, > > With Rust 1.55.0 > libc.so.7`__je_malloc_mutex_prefork+0x124 > libc.so.7`__je_arena_prefork7+0x73 > libc.so.7`_malloc_prefork+0x15b > libthr.so.3`0x392e4a8c4686 > libthr.so.3`_fork+0x18 > test-dad15ed382b075cf`nix::unistd::fork::h358225d652a86eab+0xe > test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::hb93c7cdf2b79d680+0x36 > test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::_$u7b$$u7b$closure$u7d$$u7d$::h329a121974ff9291+0x11 > test-dad15ed382b075cf`core::ops::function::FnOnce::call_once::h2261827bcba63036+0x11 > test-dad15ed382b075cf`test::__rust_begin_short_backtrace::hefb7644d11da2ff9+0xa > test-dad15ed382b075cf`test::run_test::run_test_inner::_$u7b$$u7b$closure$u7d$$u7d$::hdaa0fb71aac8d97e+0x2f3 > test-dad15ed382b075cf`std::sys_common::backtrace::__rust_begin_short_backtrace::h8bcc057a546c1087+0xce > test-dad15ed382b075cf`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf7d978d08be459d0+0x6a > test-dad15ed382b075cf`std::sys::unix::thread::Thread::new::thread_start::h6b52ca0eca213387+0x2b What are the identifiers at the end of each function? > libthr.so.3`0x392e4a8c3a7a > > With Rust 1.56.0 > libc.so.7`__je_malloc_mutex_prefork+0x124 > libc.so.7`__je_arena_prefork7+0x73 > libc.so.7`_malloc_prefork+0x15b > libthr.so.3`0x1106cebc6686 > libthr.so.3`_fork+0x18 > test-b377ad62cc9e0624`nix::unistd::fork::hbf1ed55b658aa870+0xa > 0x8 > 0xcccccccccccccccc > > See? dtrace still prints the C part of the stack, but it only prints > one or sometimes two frames of the Rust stack. ustack() unwinds the stack using the frame pointer saved in each stack frame. It'll fail to unwind code compiled without a frame pointer, e.g., if -fomit-frame-pointer is specified to a C/C++ compiler. For this and similar reasons, we compile the base system with -fno-omit-frame-pointer by default. > I'm not a compiler guy, so I don't know how to fix it. I don't even > know if the problem lies in Rust or dtrace. Would any of you smart > people be able to help here? This is a pretty important feature for > Rust development. I'd guess that Rust started omitting use of the frame pointer in generated code. This commit seems to indicate that that's the case: https://github.com/rust-lang/rust/pull/48786/commits/5b800c231f45fcd823a3e958bb942cd620ceb3e0 Though, it's rather old. I'm not sure why the problem appeared only now. So maybe the problem is elsewhere, but the commit log also mentions a -Cforce-frame-pointers flag that you could try. While it's possible to unwind the stack without using a frame pointer, a frame pointer makes doing so dead simple. ustack() does the unwinding in the kernel, in DTrace probe context, and text addresses are resolved to symbol names in userspace. So it's generally desirable to keep the implementation simple.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Ypkjrmmm%2BK34vR00>