From nobody Thu Jun 2 21:27:48 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4B3B01B66BBC for ; Thu, 2 Jun 2022 21:28:07 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LDfJ63HGpz4jgw; Thu, 2 Jun 2022 21:28:06 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-d39f741ba0so8339598fac.13; Thu, 02 Jun 2022 14:28:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mGcdg+Fzmvs+82dakUUKCOsbML853D+OtgftCPAyywM=; b=HwP6qdlKaPGmVqtEPehPhZtHIdbVDz+SALJnIACRwfOgUza0f07ptKbH0ZKdfwxeL0 vWExf99xqrLRJE/UCM6M+sT6673BSAjSDkgp90z0vk6IL/vzHJMAYpiisi+hVfgbrRfE Rf5z16qsoM/Pq60rY3pEa4WpXzdHT3HbYaTPg7ppfUC/RY2swOEg/yqhDZ6iXYhh49qF 0x62IUBYSzyaUZlq4oL8p1C+tpc3Nsl2ODnezCS/RV60GXoRIdF0nX80wH4jL1bVnwEO r07xSprpm3o4OG4dCsTdT7d0s0a8M3FAcf1+T2D/V87f9l4/dK/8ABgKqCdc2y9RBgEL 4veQ== X-Gm-Message-State: AOAM532VqNY+tgZvQ+c3g12rcMyLM6eIablOwoC1QdD0WitOVUnDnPa5 +ODDOX4hIHAzUPvP9jCPegULGIL51MJoZcNA0R9+Kzmt X-Google-Smtp-Source: ABdhPJxHdUIvFmh3b2vlXrPl0mA1FIh/e4xIdlwNg4s/SjuWdC1icAUeWxuaWrZV2naq9sVeRtj10QazdS6TBi9eVU4= X-Received: by 2002:a05:6870:1787:b0:f2:b7a9:4ea7 with SMTP id r7-20020a056870178700b000f2b7a94ea7mr20464651oae.222.1654205279457; Thu, 02 Jun 2022 14:27:59 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Thu, 2 Jun 2022 15:27:48 -0600 Message-ID: Subject: Re: Dtrace, Rust, and LLVM13 To: Mark Johnston Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4LDfJ63HGpz4jgw X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.160.50 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; FREEFALL_USER(0.00)[asomers]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.160.50:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_TLS_ALL(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.50:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On Thu, Jun 2, 2022 at 2:55 PM Mark Johnston wrote: > > On Wed, Jun 01, 2022 at 09:15:28PM -0600, Alan Somers wrote: > > The best way to profile a Rust program on FreeBSD is with Brendan > > Gregg's Flamegraph[1]. This is based on dtrace's ustack() function. > > It used to work great. However, Rust v1.56.0 is based on LLVM13[2] > > and now dtrace can't print user stacks anymore. For example, > > > > With Rust 1.55.0 > > libc.so.7`__je_malloc_mutex_prefork+0x124 > > libc.so.7`__je_arena_prefork7+0x73 > > libc.so.7`_malloc_prefork+0x15b > > libthr.so.3`0x392e4a8c4686 > > libthr.so.3`_fork+0x18 > > test-dad15ed382b075cf`nix::unistd::fork::h358225d652a86eab+0xe > > test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::hb93c7cdf2b79d680+0x36 > > test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::_$u7b$$u7b$closure$u7d$$u7d$::h329a121974ff9291+0x11 > > test-dad15ed382b075cf`core::ops::function::FnOnce::call_once::h2261827bcba63036+0x11 > > test-dad15ed382b075cf`test::__rust_begin_short_backtrace::hefb7644d11da2ff9+0xa > > test-dad15ed382b075cf`test::run_test::run_test_inner::_$u7b$$u7b$closure$u7d$$u7d$::hdaa0fb71aac8d97e+0x2f3 > > test-dad15ed382b075cf`std::sys_common::backtrace::__rust_begin_short_backtrace::h8bcc057a546c1087+0xce > > test-dad15ed382b075cf`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf7d978d08be459d0+0x6a > > test-dad15ed382b075cf`std::sys::unix::thread::Thread::new::thread_start::h6b52ca0eca213387+0x2b > > What are the identifiers at the end of each function? You mean the hex number that comes before the offset? I think the Rust compiler uses those to disambiguate closures, which would otherwise have the same name. However, not all of these functions are closures. Maybe Rust always generates that identifier anyway? > > > libthr.so.3`0x392e4a8c3a7a > > > > With Rust 1.56.0 > > libc.so.7`__je_malloc_mutex_prefork+0x124 > > libc.so.7`__je_arena_prefork7+0x73 > > libc.so.7`_malloc_prefork+0x15b > > libthr.so.3`0x1106cebc6686 > > libthr.so.3`_fork+0x18 > > test-b377ad62cc9e0624`nix::unistd::fork::hbf1ed55b658aa870+0xa > > 0x8 > > 0xcccccccccccccccc > > > > See? dtrace still prints the C part of the stack, but it only prints > > one or sometimes two frames of the Rust stack. > > ustack() unwinds the stack using the frame pointer saved in each stack > frame. It'll fail to unwind code compiled without a frame pointer, > e.g., if -fomit-frame-pointer is specified to a C/C++ compiler. For > this and similar reasons, we compile the base system with > -fno-omit-frame-pointer by default. > > > I'm not a compiler guy, so I don't know how to fix it. I don't even > > know if the problem lies in Rust or dtrace. Would any of you smart > > people be able to help here? This is a pretty important feature for > > Rust development. > > I'd guess that Rust started omitting use of the frame pointer in > generated code. This commit seems to indicate that that's the case: > https://github.com/rust-lang/rust/pull/48786/commits/5b800c231f45fcd823a3e958bb942cd620ceb3e0 > Though, it's rather old. I'm not sure why the problem appeared only > now. So maybe the problem is elsewhere, but the commit log also > mentions a -Cforce-frame-pointers flag that you could try. > > While it's possible to unwind the stack without using a frame pointer, a > frame pointer makes doing so dead simple. ustack() does the unwinding > in the kernel, in DTrace probe context, and text addresses are resolved > to symbol names in userspace. So it's generally desirable to keep the > implementation simple. That works! Setting RUSTFLAGS="-C force-frame-pointers" allows dtrace to unwind the stacks in the resulting binaries. However, it still doesn't work when I build a Rust shared library. Any idea why that might be?