From nobody Thu Jun 2 20:55:10 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BD9AC1B5E6CE for ; Thu, 2 Jun 2022 20:55:15 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LDdZC0X2Mz4c5r; Thu, 2 Jun 2022 20:55:15 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x72b.google.com with SMTP id c144so2081375qkg.11; Thu, 02 Jun 2022 13:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=niDxoMOEs2hwUkqQVVLEAV0MGA6gs+ZqeNxIKtHjQgI=; b=cjVXNgZHk5tvkrFLzGN9UNoXSGp4GSRIxkIxPTpczXOHupFTFXK0d528Qs3gAXow2/ EG199XETKNYbLic5XoMFuDMIpmF0DMWo752DPPgoA+x+HnrulgXqgjcvb5vSRpPtglcF iAJ/L/FaEYferZGy53iiSCWqUGReFoMQZ9nVtJhAHTao/5spB6guAMf4KtsCFuwpuJj/ CK3PonsXdPqubWiWVAd+p5909txcM9YtTaT0nbZbiQP2dFDeqF/2zYtqhjePR8xeAZ94 6oyNljdz3cKqLH5ZGLgZtoIs35T1fDDkp0p5LEcrOwgNNZ9qwyQnmiiYjXTaTcynpWPu lnwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=niDxoMOEs2hwUkqQVVLEAV0MGA6gs+ZqeNxIKtHjQgI=; b=qlcil3owmDtjplYl8P+iPzZmFNUF0nG10jULeQLnkHQewHowpniZ9we23/EEwj471J b0G+rrYiEzjUD4q8yehG6+C9gobe3JTlIsQoXju5O28PtjANTOcpgdFL+awqJRoYhRmP gLTCIKqWUxis6zdDjxe661DL16pSbKCSjY0ufbEJPPnvAJlm7L7L9DaZnqxqEPRUmqtU PMbCZLCF+7UgRu/x4KYvkXLkcWdnvdIFf0z9bk5vhvG6nPyUO5VwNXTxAzK+g0gMj48K hSy1ZBttREj2YB+tWw+oNM+0peRr8PQFUaydvF2YCUvRLQS3ldRE//SHDRijz1vligdc J3Ow== X-Gm-Message-State: AOAM530X4z8jGCixFuoWs0vVoJrgAxJgLD5CPQWXRoO0HGqVSVfjBUcr fQX7urgkossfmzBvMG7iNKmKt71Rz1Y= X-Google-Smtp-Source: ABdhPJyG0VwyAAqfAKFo+fOdbxiAvshlUjElrVP6g1NbkedarKr+lyFZwAkf3fbIeCH5WPhttrk7LA== X-Received: by 2002:a05:620a:b89:b0:6a3:4c5d:375d with SMTP id k9-20020a05620a0b8900b006a34c5d375dmr4478054qkh.249.1654203314137; Thu, 02 Jun 2022 13:55:14 -0700 (PDT) Received: from nuc (198-84-189-58.cpe.teksavvy.com. [198.84.189.58]) by smtp.gmail.com with ESMTPSA id o17-20020a05622a139100b002f90517bc25sm3691123qtk.41.2022.06.02.13.55.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jun 2022 13:55:13 -0700 (PDT) Date: Thu, 2 Jun 2022 16:55:10 -0400 From: Mark Johnston To: Alan Somers Cc: FreeBSD Hackers Subject: Re: Dtrace, Rust, and LLVM13 Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4LDdZC0X2Mz4c5r X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=cjVXNgZH; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-1.45 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_SHORT(0.25)[0.252]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72b:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On Wed, Jun 01, 2022 at 09:15:28PM -0600, Alan Somers wrote: > The best way to profile a Rust program on FreeBSD is with Brendan > Gregg's Flamegraph[1]. This is based on dtrace's ustack() function. > It used to work great. However, Rust v1.56.0 is based on LLVM13[2] > and now dtrace can't print user stacks anymore. For example, > > With Rust 1.55.0 > libc.so.7`__je_malloc_mutex_prefork+0x124 > libc.so.7`__je_arena_prefork7+0x73 > libc.so.7`_malloc_prefork+0x15b > libthr.so.3`0x392e4a8c4686 > libthr.so.3`_fork+0x18 > test-dad15ed382b075cf`nix::unistd::fork::h358225d652a86eab+0xe > test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::hb93c7cdf2b79d680+0x36 > test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::_$u7b$$u7b$closure$u7d$$u7d$::h329a121974ff9291+0x11 > test-dad15ed382b075cf`core::ops::function::FnOnce::call_once::h2261827bcba63036+0x11 > test-dad15ed382b075cf`test::__rust_begin_short_backtrace::hefb7644d11da2ff9+0xa > test-dad15ed382b075cf`test::run_test::run_test_inner::_$u7b$$u7b$closure$u7d$$u7d$::hdaa0fb71aac8d97e+0x2f3 > test-dad15ed382b075cf`std::sys_common::backtrace::__rust_begin_short_backtrace::h8bcc057a546c1087+0xce > test-dad15ed382b075cf`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf7d978d08be459d0+0x6a > test-dad15ed382b075cf`std::sys::unix::thread::Thread::new::thread_start::h6b52ca0eca213387+0x2b What are the identifiers at the end of each function? > libthr.so.3`0x392e4a8c3a7a > > With Rust 1.56.0 > libc.so.7`__je_malloc_mutex_prefork+0x124 > libc.so.7`__je_arena_prefork7+0x73 > libc.so.7`_malloc_prefork+0x15b > libthr.so.3`0x1106cebc6686 > libthr.so.3`_fork+0x18 > test-b377ad62cc9e0624`nix::unistd::fork::hbf1ed55b658aa870+0xa > 0x8 > 0xcccccccccccccccc > > See? dtrace still prints the C part of the stack, but it only prints > one or sometimes two frames of the Rust stack. ustack() unwinds the stack using the frame pointer saved in each stack frame. It'll fail to unwind code compiled without a frame pointer, e.g., if -fomit-frame-pointer is specified to a C/C++ compiler. For this and similar reasons, we compile the base system with -fno-omit-frame-pointer by default. > I'm not a compiler guy, so I don't know how to fix it. I don't even > know if the problem lies in Rust or dtrace. Would any of you smart > people be able to help here? This is a pretty important feature for > Rust development. I'd guess that Rust started omitting use of the frame pointer in generated code. This commit seems to indicate that that's the case: https://github.com/rust-lang/rust/pull/48786/commits/5b800c231f45fcd823a3e958bb942cd620ceb3e0 Though, it's rather old. I'm not sure why the problem appeared only now. So maybe the problem is elsewhere, but the commit log also mentions a -Cforce-frame-pointers flag that you could try. While it's possible to unwind the stack without using a frame pointer, a frame pointer makes doing so dead simple. ustack() does the unwinding in the kernel, in DTrace probe context, and text addresses are resolved to symbol names in userspace. So it's generally desirable to keep the implementation simple.