From owner-freebsd-net@FreeBSD.ORG Mon Mar 17 17:40:01 2014 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62A9473C for ; Mon, 17 Mar 2014 17:40:01 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4EEFE138 for ; Mon, 17 Mar 2014 17:40:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2HHe0Q9035534 for ; Mon, 17 Mar 2014 17:40:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2HHe08t035533; Mon, 17 Mar 2014 17:40:00 GMT (envelope-from gnats) Date: Mon, 17 Mar 2014 17:40:00 GMT Message-Id: <201403171740.s2HHe08t035533@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: "Charbon, Julien" Subject: Re: kern/183659: [tcp] TCP stack lock contention with short-lived connections X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: "Charbon, Julien" List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2014 17:40:01 -0000 The following reply was made to PR kern/183659; it has been noted by GNATS. From: "Charbon, Julien" To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/183659: [tcp] TCP stack lock contention with short-lived connections Date: Mon, 17 Mar 2014 15:54:03 +0100 Just a follow-up that updates lock profiling results with short-lived TCP connection traffic on FreeBSD-10.0 RELEASE: (Previous results were made on FreeBSD-9.2 RELEASE) o FreeBSD-10 RELEASE: # sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -5 debug.lock.prof.stats: max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name 37 321900 3049892 13033648 610019 4 21 0 588013 sys/netinet/tcp_input.c:778 (rw:tcp) tcp_input() (SYN|FIN|RST) 51 115462 3240265 12270984 553157 5 22 0 545293 sys/netinet/tcp_input.c:1013 (rw:tcp) tcp_input() (state != ESTABLISHED) 29 62577 1170617 8754815 305885 3 28 0 296845 sys/netinet/tcp_usrreq.c:728 (rw:tcp) tcp_usr_close() 6 62645 146544 8548857 292058 0 29 0 283587 sys/netinet/tcp_usrreq.c:984 (rw:tcp) tcp_usr_shutdown() 11 62595 198811 6525067 309009 0 21 0 304522 sys/netinet/tcp_usrreq.c:635 (rw:tcp) tcp_usr_accept() - If lock contention spots moved a little between 9.2 and 10.0, nothing major as the top 5 still belongs to (rw:tcp) lock (a.k.a. TCP INP_INFO). o FreeBSD-10 RELEASE + PCBGROUP kernel option (by popular demand): # sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -5 debug.lock.prof.stats: max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name 58 84250 2970633 13154832 622401 4 21 0 598964 sys/netinet/tcp_input.c:778 (rw:tcp) tcp_input() (SYN|FIN|RST) 47 224326 3375328 12945466 562451 6 23 0 554567 sys/netinet/tcp_input.c:1013 (rw:tcp) tcp_input() (state != ESTABLISHED) 22 84332 1193078 9693951 311555 3 31 0 302420 sys/netinet/tcp_usrreq.c:728 (rw:tcp) tcp_usr_close() 6 84307 151411 9137383 298120 0 30 0 289496 sys/netinet/tcp_usrreq.c:984 (rw:tcp) tcp_usr_shutdown() 15 84351 201705 6504520 314353 0 20 0 310270 sys/netinet/tcp_usrreq.c:635 (rw:tcp) tcp_usr_accept() - No changes at all in first ranks by using PCBGROUP option on FreeBSD-10 RELEASE. I have indeed checked that PCBGROUP was in use as at #36 rank there is the specific pcbgroup lock: 11 9 289817 4815 1505626 0 0 0 16054 sys/netinet/in_pcb.c:1530 (sleep mutex:pcbgroup) o FreeBSD-10 RELEASE + current lock mitigation patches [1][2]: # sysctl debug.lock.prof.stats | head -2; sysctl debug.lock.prof.stats | sort -n -k 4 -r | head -20 debug.lock.prof.stats: max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name 29 297 3781629 13476466 734686 5 18 0 715214 sys/netinet/tcp_input.c:778 (rw:tcp) tcp_input() (SYN|FIN|RST) 35 287 3817278 12301410 672907 5 18 0 669324 sys/netinet/tcp_input.c:1013 (rw:tcp) tcp_input() (state != ESTABLISHED) 18 170 1392058 2494823 367131 3 6 0 357888 sys/netinet/tcp_usrreq.c:719 (rw:tcp) tcp_usr_shutdown() 7 141 182209 2433120 350488 0 6 0 344878 sys/netinet/tcp_usrreq.c:975 (rw:tcp) tcp_usr_close() 10 259 26786 933073 38101 0 24 0 37624 sys/netinet/tcp_timer.c:493 (rw:tcp) tcp_timer_rexmt() - No more tcp_usr_accept() (expected) o Global results: Maximum short-lived TCP connection rate without dropping a single packet: - FreeBSD 10.0 RELEASE: 40.0k - FreeBSD 10.0 RELEASE + PCBGROUP: 40.0k - FreeBSD 10.0 RELEASE + patches: 56.8k [1] Decrease lock contention within the TCP accept case by removing the INP_INFO lock from tcp_usr_accept. http://svnweb.freebsd.org/base?view=revision&revision=261242 [2] tw-clock-v2.patch attached in: http://lists.freebsd.org/pipermail/freebsd-net/2014-March/038124.html -- Julien