Date: Mon, 22 Oct 2018 14:45:19 +0300 From: Lev Serebryakov <lev@FreeBSD.org> To: Alan Somers <asomers@freebsd.org> Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: What is wrong with dtrace's stack()? Message-ID: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> In-Reply-To: <CAOtMX2goztCYJXM72WNR711DEBEKAGk1zFHF=NpE4UiVAF2bAQ@mail.gmail.com> References: <170994671.20181021201021@serebryakov.spb.ru> <CAG6CVpVCe-8OVU%2BKEyAh8swFSAjRshNcNLN0tjAS3W8bZxyrTA@mail.gmail.com> <475670271.20181022003734@serebryakov.spb.ru> <CAOtMX2goztCYJXM72WNR711DEBEKAGk1zFHF=NpE4UiVAF2bAQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd
Content-Type: multipart/mixed; boundary="22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T";
protected-headers="v1"
From: Lev Serebryakov <lev@FreeBSD.org>
Reply-To: lev@FreeBSD.org
To: Alan Somers <asomers@freebsd.org>
Cc: cse.cem@gmail.com,
"freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
Message-ID: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org>
Subject: Re: What is wrong with dtrace's stack()?
References: <170994671.20181021201021@serebryakov.spb.ru>
<CAG6CVpVCe-8OVU+KEyAh8swFSAjRshNcNLN0tjAS3W8bZxyrTA@mail.gmail.com>
<475670271.20181022003734@serebryakov.spb.ru>
<CAOtMX2goztCYJXM72WNR711DEBEKAGk1zFHF=NpE4UiVAF2bAQ@mail.gmail.com>
In-Reply-To: <CAOtMX2goztCYJXM72WNR711DEBEKAGk1zFHF=NpE4UiVAF2bAQ@mail.gmail.com>
--22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 22.10.2018 0:47, Alan Somers wrote:
> =C2=A0I see a lot of stacks like this:
>=20
> =C2=A0kernel`lock_delay+0x42
> =C2=A0kernel`soo_write+0x33
> =C2=A0kernel`dofilewrite+0x79
> =C2=A0kernel`sys_write+0xc3
> =C2=A0kernel`amd64_syscall+0x332
> =C2=A0kernel`0xffffffff8086c87d
>=20
> =C2=A0But event sosend() doesn't call lock_delay(), so it is imposs=
ible to
> understand why do lock_delay() seen 41932 times in 60 seconds at to=
p
> of the
> stack. Where are all call stack?! All these functions could not be
> inlined,
> as sosend() is located in other translation unit and it calls
> function by
> pointer, this call could not be inlined too.
>=20
>=20
> If you're sure that the function isn't inlined, then it might be using
> the tail-call optimization instead.=C2=A0 That would also explain the m=
issing
> stack frames, too.
I know about inlining at TCO, but it is not the case for sure. Problem
is, they can not be inlined it TCO'ed.
soo_wrtite() calls sosend():
error =3D sosend(so, 0, uio, 0, 0, 0, uio->uio_td);
if (error =3D=3D EPIPE && (so->so_options & SO_NOSIGPIPE) =3D=3D 0) {
PROC_LOCK(uio->uio_td->td_proc);
tdsignal(uio->uio_td, SIGPIPE);
PROC_UNLOCK(uio->uio_td->td_proc);
}
It could not be TCO'ed and sosend() is in another translation unit, so
it could not be inlined.
sosend() calls protocol-specific handler via function pointer, so it
should be true call:
CURVNET_SET(so->so_vnet);
if (!SOLISTENING(so))
error =3D so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio,
top, control, flags, td);
else {
m_freem(top);
m_freem(control);
error =3D ENOTCONN;
}
CURVNET_RESTORE();
return (error);
These frames MUST be here...
> If you can manually narrow the options down to a few
> possible callers, then you could try adding a few SDT probes.=C2=A0 Tha=
t's
> what I usually do in cases like this.=C2=A0 Or, for static functions th=
at are
> inlined, you can remove the static keyword to (usually) prevent the
> inlining.
Narrowing all paths, which leads to very generic lock_delay()... Uh-oh.=
--=20
// Lev Serebryakov
--22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T--
--6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvNuFBfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5
NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c
R4+33A/+P+bUbYn+N6DMGvCNsJN89TmU4V/TbEZVwIFMDRw+Lkoygf0m2/Q9ts1O
8eFmOGtCBEryiUIfp8DG9XsOxCal/AH+4JNy5C52cXY/lwfZRVYwltBfBm+0DzQC
+coHrc6IA9aRvMQc04esZ6wf/GMY47mN1fYc5kr6jou74BHe//VVnWcIPQ1NumHK
5MEGUr+dD7eNWH6gMv0YkwU/mIK7ebz1rscdjUlAjba5l1r8PU8lR6MWl7cuUb1t
7Rec5waJ4E9LoayPojyTIQvKx0rELpML+w0wlCCnqpuEn8NcpvAe9cCkCipc68K4
Vx0fGN+1MQRRab0FkTXvpR+TeiWskDaLktO/o2QIY5rac+6xeqVHzSncY27ydkNj
x7F39Swh2Yr217h6gS68iMPjf+N2Jn/+IolLiCacXSY3lwd7qnc0YZnWV6Rxnwqo
CnSgRlV5OMt1M8iCKiko0G3fpEdPOwlt1BTha0mL6zh4ybRvf8AjZvBHqDKmvClj
bwzARN0ZaAqu2EtRf8Cf43zRdrNeHTwesYn13l66n60AuoWA48aHclgu7Vqp/Gzs
eF51WUuiLB4g865s4KRyyy++g0fS0WQGnx8YeP+df7zJQ0TUC2pYRUqu9OTdQy2t
/yqtMqtMjbKbT3wU0uJX19EWCQS9FY5nvztoKUt5S5chl/PZRuQ=
=/2k9
-----END PGP SIGNATURE-----
--6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?234d745d-37a9-9610-15b9-0f5cd5af21bf>