From owner-freebsd-hackers@freebsd.org Mon Oct 22 11:45:28 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A253FF80E0 for ; Mon, 22 Oct 2018 11:45:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id A964175FFE; Mon, 22 Oct 2018 11:45:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 790EA79E; Mon, 22 Oct 2018 14:45:26 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Alan Somers Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> Date: Mon, 22 Oct 2018 14:45:19 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2018 11:45:28 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd Content-Type: multipart/mixed; boundary="22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Alan Somers Cc: cse.cem@gmail.com, "freebsd-hackers@freebsd.org" Message-ID: <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> In-Reply-To: --22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 22.10.2018 0:47, Alan Somers wrote: > =C2=A0I see a lot of stacks like this: >=20 > =C2=A0kernel`lock_delay+0x42 > =C2=A0kernel`soo_write+0x33 > =C2=A0kernel`dofilewrite+0x79 > =C2=A0kernel`sys_write+0xc3 > =C2=A0kernel`amd64_syscall+0x332 > =C2=A0kernel`0xffffffff8086c87d >=20 > =C2=A0But event sosend() doesn't call lock_delay(), so it is imposs= ible to > understand why do lock_delay() seen 41932 times in 60 seconds at to= p > of the > stack. Where are all call stack?! All these functions could not be > inlined, > as sosend() is located in other translation unit and it calls > function by > pointer, this call could not be inlined too. >=20 >=20 > If you're sure that the function isn't inlined, then it might be using > the tail-call optimization instead.=C2=A0 That would also explain the m= issing > stack frames, too. I know about inlining at TCO, but it is not the case for sure. Problem is, they can not be inlined it TCO'ed. soo_wrtite() calls sosend(): error =3D sosend(so, 0, uio, 0, 0, 0, uio->uio_td); if (error =3D=3D EPIPE && (so->so_options & SO_NOSIGPIPE) =3D=3D 0) { PROC_LOCK(uio->uio_td->td_proc); tdsignal(uio->uio_td, SIGPIPE); PROC_UNLOCK(uio->uio_td->td_proc); } It could not be TCO'ed and sosend() is in another translation unit, so it could not be inlined. sosend() calls protocol-specific handler via function pointer, so it should be true call: CURVNET_SET(so->so_vnet); if (!SOLISTENING(so)) error =3D so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, top, control, flags, td); else { m_freem(top); m_freem(control); error =3D ENOTCONN; } CURVNET_RESTORE(); return (error); These frames MUST be here... > If you can manually narrow the options down to a few > possible callers, then you could try adding a few SDT probes.=C2=A0 Tha= t's > what I usually do in cases like this.=C2=A0 Or, for static functions th= at are > inlined, you can remove the static keyword to (usually) prevent the > inlining. Narrowing all paths, which leads to very generic lock_delay()... Uh-oh.= --=20 // Lev Serebryakov --22oBgQaHz796XrnDIOOx5mFOGHtuyBc1T-- --6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvNuFBfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4+33A/+P+bUbYn+N6DMGvCNsJN89TmU4V/TbEZVwIFMDRw+Lkoygf0m2/Q9ts1O 8eFmOGtCBEryiUIfp8DG9XsOxCal/AH+4JNy5C52cXY/lwfZRVYwltBfBm+0DzQC +coHrc6IA9aRvMQc04esZ6wf/GMY47mN1fYc5kr6jou74BHe//VVnWcIPQ1NumHK 5MEGUr+dD7eNWH6gMv0YkwU/mIK7ebz1rscdjUlAjba5l1r8PU8lR6MWl7cuUb1t 7Rec5waJ4E9LoayPojyTIQvKx0rELpML+w0wlCCnqpuEn8NcpvAe9cCkCipc68K4 Vx0fGN+1MQRRab0FkTXvpR+TeiWskDaLktO/o2QIY5rac+6xeqVHzSncY27ydkNj x7F39Swh2Yr217h6gS68iMPjf+N2Jn/+IolLiCacXSY3lwd7qnc0YZnWV6Rxnwqo CnSgRlV5OMt1M8iCKiko0G3fpEdPOwlt1BTha0mL6zh4ybRvf8AjZvBHqDKmvClj bwzARN0ZaAqu2EtRf8Cf43zRdrNeHTwesYn13l66n60AuoWA48aHclgu7Vqp/Gzs eF51WUuiLB4g865s4KRyyy++g0fS0WQGnx8YeP+df7zJQ0TUC2pYRUqu9OTdQy2t /yqtMqtMjbKbT3wU0uJX19EWCQS9FY5nvztoKUt5S5chl/PZRuQ= =/2k9 -----END PGP SIGNATURE----- --6bbKwImKb5FFJkLwlt8YbdllYL69mCCwd--