Date: Mon, 17 Mar 2014 17:40:00 GMT From: "Charbon, Julien" <jcharbon@verisign.com> To: freebsd-net@FreeBSD.org Subject: Re: kern/183659: [tcp] TCP stack lock contention with short-lived connections Message-ID: <201403171740.s2HHe08t035533@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/183659; it has been noted by GNATS.
From: "Charbon, Julien" <jcharbon@verisign.com>
To: bug-followup@FreeBSD.org
Cc:
Subject: Re: kern/183659: [tcp] TCP stack lock contention with short-lived
connections
Date: Mon, 17 Mar 2014 15:54:03 +0100
Just a follow-up that updates lock profiling results with
short-lived TCP connection traffic on FreeBSD-10.0 RELEASE:
(Previous results were made on FreeBSD-9.2 RELEASE)
o FreeBSD-10 RELEASE:
# sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -5
debug.lock.prof.stats:
max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name
37 321900 3049892 13033648 610019 4 21 0 588013 sys/netinet/tcp_input.c:778 (rw:tcp) tcp_input() (SYN|FIN|RST)
51 115462 3240265 12270984 553157 5 22 0 545293 sys/netinet/tcp_input.c:1013 (rw:tcp) tcp_input() (state != ESTABLISHED)
29 62577 1170617 8754815 305885 3 28 0 296845 sys/netinet/tcp_usrreq.c:728 (rw:tcp) tcp_usr_close()
6 62645 146544 8548857 292058 0 29 0 283587 sys/netinet/tcp_usrreq.c:984 (rw:tcp) tcp_usr_shutdown()
11 62595 198811 6525067 309009 0 21 0 304522 sys/netinet/tcp_usrreq.c:635 (rw:tcp) tcp_usr_accept()
- If lock contention spots moved a little between 9.2 and 10.0, nothing
major as the top 5 still belongs to (rw:tcp) lock (a.k.a. TCP INP_INFO).
o FreeBSD-10 RELEASE + PCBGROUP kernel option (by popular demand):
# sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -5
debug.lock.prof.stats:
max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name
58 84250 2970633 13154832 622401 4 21 0 598964 sys/netinet/tcp_input.c:778 (rw:tcp) tcp_input() (SYN|FIN|RST)
47 224326 3375328 12945466 562451 6 23 0 554567 sys/netinet/tcp_input.c:1013 (rw:tcp) tcp_input() (state != ESTABLISHED)
22 84332 1193078 9693951 311555 3 31 0 302420 sys/netinet/tcp_usrreq.c:728 (rw:tcp) tcp_usr_close()
6 84307 151411 9137383 298120 0 30 0 289496 sys/netinet/tcp_usrreq.c:984 (rw:tcp) tcp_usr_shutdown()
15 84351 201705 6504520 314353 0 20 0 310270 sys/netinet/tcp_usrreq.c:635 (rw:tcp) tcp_usr_accept()
- No changes at all in first ranks by using PCBGROUP option on
FreeBSD-10 RELEASE. I have indeed checked that PCBGROUP was in
use as at #36 rank there is the specific pcbgroup lock:
11 9 289817 4815 1505626 0 0 0 16054 sys/netinet/in_pcb.c:1530 (sleep mutex:pcbgroup)
o FreeBSD-10 RELEASE + current lock mitigation patches [1][2]:
# sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -20
debug.lock.prof.stats:
max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name
29 297 3781629 13476466 734686 5 18 0 715214 sys/netinet/tcp_input.c:778 (rw:tcp) tcp_input() (SYN|FIN|RST)
35 287 3817278 12301410 672907 5 18 0 669324 sys/netinet/tcp_input.c:1013 (rw:tcp) tcp_input() (state != ESTABLISHED)
18 170 1392058 2494823 367131 3 6 0 357888 sys/netinet/tcp_usrreq.c:719 (rw:tcp) tcp_usr_shutdown()
7 141 182209 2433120 350488 0 6 0 344878 sys/netinet/tcp_usrreq.c:975 (rw:tcp) tcp_usr_close()
10 259 26786 933073 38101 0 24 0 37624 sys/netinet/tcp_timer.c:493 (rw:tcp) tcp_timer_rexmt()
- No more tcp_usr_accept() (expected)
o Global results: Maximum short-lived TCP connection rate without dropping a single packet:
- FreeBSD 10.0 RELEASE: 40.0k
- FreeBSD 10.0 RELEASE + PCBGROUP: 40.0k
- FreeBSD 10.0 RELEASE + patches: 56.8k
[1] Decrease lock contention within the TCP accept case by removing
the INP_INFO lock from tcp_usr_accept.
http://svnweb.freebsd.org/base?view=revision&revision=261242
[2] tw-clock-v2.patch attached in:
http://lists.freebsd.org/pipermail/freebsd-net/2014-March/038124.html
--
Julien
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201403171740.s2HHe08t035533>