From owner-freebsd-hackers@freebsd.org Sun Oct 21 21:47:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6C6D1037654 for ; Sun, 21 Oct 2018 21:47:24 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CADF7A0BC; Sun, 21 Oct 2018 21:47:24 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f41.google.com with SMTP id l1-v6so9064974lfc.3; Sun, 21 Oct 2018 14:47:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cQ/MaefgTJ8AP9zpAUvm1pangjFtCx2xExMZZ5KRi7M=; b=RGAcpWz4olp1HuxLF9ri0LbaWlXXD4KhKoZcb/MV/JOAkvbujfQQ9t1+IC8OmNc+ec jBAQb2S3sLKC5fwXFNFxrDxbfHCU6GgvIQOz+UBvRRQFgZlBtlC3nsKbYHMZ+161V4p2 Wfga718TRxJtUgLN9H3vtAPXiEJDONSIJ1JEv2XECgWIgQgTZCKo2js7GH0J8bbHa/Y+ x0qU6dw9vlxCogthAg/67wQYGB9aoQK2JVLgajqCGdjnxpJ7i2jQzJVkJhhpJ41ZK6z5 nSbVDtRiMafMQIA1SnaUISoZyxB+lIe7HWT4E7tA65XbS6V0ulQDmzPawML6IDRu7Hl5 N9rw== X-Gm-Message-State: ABuFfoifmaT1HCcKMmp2B1RZCtIrk+4iWnRXuk1VflBGient9prkpEVj CP0SpsZujrifMtfJHBYzqEertfzCrUtp+Ww46NciBw== X-Google-Smtp-Source: ACcGV61hc+gIIzzYMw7Po2n0w8BqxMGaosQIfmz6V33yiHGZl1UUmu0C0eR2j6UsU6dXA9bCVJwTaLQQv+SnKS7A8VU= X-Received: by 2002:a19:964e:: with SMTP id y75-v6mr7871756lfd.58.1540158442187; Sun, 21 Oct 2018 14:47:22 -0700 (PDT) MIME-Version: 1.0 References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> In-Reply-To: <475670271.20181022003734@serebryakov.spb.ru> From: Alan Somers Date: Sun, 21 Oct 2018 15:47:10 -0600 Message-ID: Subject: Re: What is wrong with dtrace's stack()? To: Lev Serebryakov Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2018 21:47:24 -0000 On Sun, Oct 21, 2018 at 3:38 PM Lev Serebryakov wrote: > Hello Conrad, > > Sunday, October 21, 2018, 8:21:43 PM, you wrote: > > > Your assembler dump offsets are in decimal. Look for offset 0x33 = > > +51, not +33. > Problem is, sosend() is not very interesting by itself, and looks like > several layers of stack are always lost. > > I see a lot of stacks like this: > > kernel`lock_delay+0x42 > kernel`soo_write+0x33 > kernel`dofilewrite+0x79 > kernel`sys_write+0xc3 > kernel`amd64_syscall+0x332 > kernel`0xffffffff8086c87d > > But event sosend() doesn't call lock_delay(), so it is impossible to > understand why do lock_delay() seen 41932 times in 60 seconds at top of the > stack. Where are all call stack?! All these functions could not be inlined, > as sosend() is located in other translation unit and it calls function by > pointer, this call could not be inlined too. > If you're sure that the function isn't inlined, then it might be using the tail-call optimization instead. That would also explain the missing stack frames, too. If you can manually narrow the options down to a few possible callers, then you could try adding a few SDT probes. That's what I usually do in cases like this. Or, for static functions that are inlined, you can remove the static keyword to (usually) prevent the inlining. -Alan > > > -- > Best regards, > Lev mailto:lev@FreeBSD.org > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >