Date: Wed, 24 Oct 2018 18:06:53 +0300 From: Lev Serebryakov <lev@FreeBSD.org> To: freebsd-net@freebsd.org Subject: Very high lock contention in tcp_usr_send() which looks to limit if_gif performance with standard MTU Message-ID: <1ba8044b-9dea-8d22-1fc8-e15ff7b18794@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--LPSH1QUfrB0Vb7nNGnUCTBKrbYxLyGVEh
Content-Type: multipart/mixed; boundary="3qQE843I85WSOce41UXqygxSapGYHyerT";
protected-headers="v1"
From: Lev Serebryakov <lev@FreeBSD.org>
Reply-To: lev@FreeBSD.org
To: freebsd-net@freebsd.org
Message-ID: <1ba8044b-9dea-8d22-1fc8-e15ff7b18794@FreeBSD.org>
Subject: Very high lock contention in tcp_usr_send() which looks to limit
if_gif performance with standard MTU
--3qQE843I85WSOce41UXqygxSapGYHyerT
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
I've benchmarked if_gre and if_gif encapsulation on not-so-powerful
hardware. Test is very simple: single as-fast-as-possible TCP stream via
tunnel to much-more performant server over 1G physical interface.
I was surprised by very high price of encapsulation with standard MTU
of 1280 bytes. Performance in such configuration is about 45% of line
speed (440Mbit instead of 1Gbit). I can not believe, that adding one
header to each packet is SO expensive.
Also, in such case userland thread of benchmark (nuttcp or iperf3)
consumes 100% of CPU (one core) and kernel thread (if_config_tqg_0 or
if_config_tqg_1) consumes 100% of CPY (second core).
Physical interface with MTU of 1280 and switched off hardware "helpers"
shows 800Mbit/s and CPU is not-so-consumed, userland and kernel consume
about 50% + 50% of CPU, not 100% + 100%, which looks reasonable.
I've traced down half of problem: userland CPU consumption to lock
contention at "sys/netinet/tcp_usrreq.c:923":
inp =3D sotoinpcb(so);
KASSERT(inp !=3D NULL, ("tcp_usr_send: inp =3D=3D NULL"));
********INP_WLOCK(inp);********
if (inp->inp_flags & (INP_TIMEWAIT | INP_DROPPED)) {
This line consumes 75% of all time consumed by sosend(), and,
effectively, by userland, via this call chain:
kernel`sosend+0x79
kernel`soo_write+0x6b
kernel`fo_write+0x4a
kernel`dofilewrite+0xcd
kernel`kern_writev+0x79
kernel`sys_write+0x8f
75% of one core is consumed by this line.
Unfortunately, other party of this contention is not so obvious.
Flame graph without TCO could be found here:
http://lev.serebryakov.spb.ru/_sklad/gif-stacks/no-tco/gif.1280.nuttcp.s=
end.no-tco.svg
It looks very suspicious to me.
BTW, if_gre has exactly same problem.
--=20
// Lev Serebryakov
--3qQE843I85WSOce41UXqygxSapGYHyerT--
--LPSH1QUfrB0Vb7nNGnUCTBKrbYxLyGVEh
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvQio1fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5
NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c
R49tEA/+JWDnBoNhcxpEWSzxW8iuvJUScaomO1Vm/rBAI9W1Ty3FKMpm2deQiQqB
RuAGKJFvEtFiJrTKhox55Cs13n0tAP/+epLqr3up7j9y8Yek9hFGPIz8Vg6Cqv0W
fXSqqjrdg0fHZO7dYNGYNWTv9pFhgYYr+ArqIGwRW1L99n1PnjLXwDMKfqODrpDS
WUSEHcjDDK1JVV69KFRkaFFpCUI8usaHleymXSBem4MhvU0LZQdIkU8Elf6AIQAE
qJ4wyiyn2hv6ZrDuqzEEdF1AvbKClKnvGJxfLYos1U7yCFHkKLgLEfS4bx1HdQ9C
D9iyRzGSLB1eTdWModOIugzrUgYcYZMOmKbhGMqiyOi+POjH68BQY36q4K8JEd4X
4RIqHuV2uENLWgVASDyIFXqEIVPzu8R/P44ajJxOsrfsSIW4WNrSoEmsgt9SSUmR
/JGN4FxZfIJ5MJ/IyfcZjBSCBW6uDN/LsdhME9GyytkEONiCC3L8PEk8Gok2sgQp
kIWhnpteS2QCIhE7jLt/bvhpi40TMBJaOj1oq7GZnD4w8C1eE0CvGNWkBb52UJDW
Xb4kT4XXyCXUPMfxH3T25XS5Wo/H+XrhsdJa1/MRqS0pxBT7F6va9b6WaCsaftyU
9mN6mQp4Z4V8mNh6JQzZgIi95LYAIou4bCqbad3ugk8T+69a0pg=
=KcLL
-----END PGP SIGNATURE-----
--LPSH1QUfrB0Vb7nNGnUCTBKrbYxLyGVEh--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1ba8044b-9dea-8d22-1fc8-e15ff7b18794>