From owner-freebsd-hackers@freebsd.org Wed Oct 24 16:10:49 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55A40FFA9EC for ; Wed, 24 Oct 2018 16:10:49 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id CE2E26B9DA; Wed, 24 Oct 2018 16:10:48 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 985A3DF6; Wed, 24 Oct 2018 19:10:47 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: What is wrong with dtrace's stack()? To: Ryan Stone Cc: freebsd-hackers@freebsd.org, Alan Somers , Conrad Meyer References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Wed, 24 Oct 2018 19:10:41 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2018 16:10:49 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6 Content-Type: multipart/mixed; boundary="12uJTI8sAMiThSkLuym1l9QItCGS2mjQC"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Ryan Stone Cc: freebsd-hackers@freebsd.org, Alan Somers , Conrad Meyer Message-ID: Subject: Re: What is wrong with dtrace's stack()? References: <170994671.20181021201021@serebryakov.spb.ru> <475670271.20181022003734@serebryakov.spb.ru> <234d745d-37a9-9610-15b9-0f5cd5af21bf@FreeBSD.org> <168122586.20181024003412@serebryakov.spb.ru> In-Reply-To: --12uJTI8sAMiThSkLuym1l9QItCGS2mjQC Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 24.10.2018 18:48, Ryan Stone wrote: > sosend_generic+0x112c is the return address, so it's one instruction > after the call instruction that called lock_delay. Yes, but this instruction (+0x112b) calls real protocol send function via function pointer. > What's the line number of sosend_generic+0x112b? It is here, uipc_socket.c:1589: error =3D (*so->so_proto->pr_usrreqs->pru_send)(so, (flags & MSG_OOB) ? PRUS_OOB : /* * If the user set MSG_EOF, the protocol understands * this flag and nothing left to send then use * PRU_SEND_EOF instead of PRU_SEND. */ ((flags & MSG_EOF) && (so->so_proto->pr_flags & PR_IMPLOPCL) && (resid <=3D 0)) ? PRUS_EOF : /* If there is more to send set PRUS_MORETOCOME. */ (flags & MSG_MORETOCOME) || (resid > 0 && space > 0) ? PRUS_MORETOCOME : 0, top, addr, control, td); I have these three DIFFERENT stack, which should be one for sure, as it is confirmed by custom SDT probes now (I've added them and checked). Addresses are different from previous examples due to some code was shifted by SDT probes. 26462 times stack is Ok, but 4068+3993 times some frames are lost. Same address could not call both tcp_usr_send() and ia32_pause() and lock_delay(). Really, it should be one stack trace! kernel`lock_delay+0x72 kernel`sosend_generic+0xf61 kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cedfbd 3993 kernel`ia32_pause+0x7 kernel`sosend_generic+0xf61 kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cedfbd 4068 kernel`ia32_pause+0x6 kernel`tcp_usr_send+0x131 kernel`sosend_generic+0xf61 kernel`sosend+0x79 kernel`soo_write+0x6b kernel`fo_write+0x4a kernel`dofilewrite+0xcd kernel`kern_writev+0x79 kernel`sys_write+0x8f kernel`syscallenter+0x774 kernel`amd64_syscall+0x1b kernel`0xffffffff80cedfbd 26462 > On Tue, Oct 23, 2018 at 5:34 PM Lev Serebryakov wrote= : >> >> Hello Ryan, >> >> Monday, October 22, 2018, 11:50:29 PM, you wrote: >> >>> Adding -fno-optimize-sibling-calls to the compiler flags will elimina= te the TCO. >> Stacks are better but still strange. For example I have such stack (w= hich I >> like better than previous): >> >> kernel`lock_delay+0x72 >> kernel`sosend_generic+0x112c >> kernel`sosend+0x79 >> kernel`soo_write+0x6b >> kernel`fo_write+0x4a >> kernel`dofilewrite+0xcd >> kernel`kern_writev+0x79 >> kernel`sys_write+0x8f >> kernel`syscallenter+0x774 >> kernel`amd64_syscall+0x1b >> kernel`0xffffffff80cebf6d >> >> According to addr2line `sosend_generic+0x112c' is >> >> https://svnweb.freebsd.org/base/head/sys/kern/uipc_socket.c?revision=3D= 339419&view=3Dmarkup#l1579 >> >> Which is call to protocol-specific send function. Where is this funct= ion >> (it should be tcp for sure)?! >> >> -- >> Best regards, >> Lev mailto:lev@FreeBSD.org >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 --=20 // Lev Serebryakov --12uJTI8sAMiThSkLuym1l9QItCGS2mjQC-- --qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvQmYFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R48MaxAAyiUF7zIrHfmfyvijJJKVBzZk8dL+ko0sUuqonaHaP/BLk088KAabpKd+ 4nqnolCcpRlmenenR54t6EvDKXAkhKvH4Qy52NAjU699uzqBDcnk4FKPpq2gKZbE Dz7WUb8itARTeq7ztFoJ00VPsLZSFoAWipuhpwvDwM1U/J54Iprh7v3rYlW7LJNJ 88Cd/xsxjt9ZtNw/F6YbncsgAIrDxSSTdAUa2oztcHoyM5x7PqrsoXcjQ8Ln2Pir ZNUFd8vI62z44IVHnuNRX/1mmWRBHFRkgnkfTKEimJIcK9ZfAccP4bS6OEJBYLLA 0c/xougcb7JJ58EWLXWQk1qzVxozGapWLozpWADSgc5AYi1g3xYlyA8Z9Qyy0emF InmKWnGHe7JPYPd5VnyI8hABH3PbQKuelmfEKgytjDrhxKXL+EWupgw20MLJhZcG 2VKEvaenoYVGYUUhhPvZ10PMQ69ulmWelC2J4n1xOYNdFLa/M3/98re8RhySuci2 bd7uk/7UTTDFIl7/y1r06Nnwjqe6ozo6ooiIHd6U0y/54iXpHO+rEKkBikTrPFF6 tHSt+8rMoP0OKq/V2U7QTulLFDPnEaanFS1Z06tw1wAVnqL3vGHdoFpcxPY4MoYZ a9w59fSX7iS4LAeUdlrwM5FMCGUx3gr5pYVJznzVEjOjOebrRTU= =KiLK -----END PGP SIGNATURE----- --qAxpfnw6JzFvUEbu8htvvBpEoe9xyZ1h6--