From owner-freebsd-net@FreeBSD.ORG Tue Dec 31 07:35:38 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E5B0CD6 for ; Tue, 31 Dec 2013 07:35:38 +0000 (UTC) Received: from mail-qa0-x230.google.com (mail-qa0-x230.google.com [IPv6:2607:f8b0:400d:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 401811A32 for ; Tue, 31 Dec 2013 07:35:38 +0000 (UTC) Received: by mail-qa0-f48.google.com with SMTP id w5so11493390qac.14 for ; Mon, 30 Dec 2013 23:35:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=1c5dJAbDe1r9gnMm2/fZvbSvu/N3QR4Ry4SbwKlneIA=; b=oEk+izeuGP6E44wKd3NjstQf8hZDpOQkYTq98RtOz9cKI8xoWUxaaam2mV2mBXqZoz s3wKdcJ93tIOA+9SR57I265kGRRYZa8DOnY7UXBkQTHP2jb9WIhkrWYIfCl+NGVVqx1d SFha9TXbn4eCj25fan0euVwIp0qhnBiLD1Z7Tz6Ob0MoKgmplRQOFxQKiPmZREAc7eST lKNBBQuhuVvjmLbapGKnvdmS66nCk3M2lUl3VcjeyOei5mYRk1X1XFhTamGlq6if45EC ZRvhJRVMRcUN/t94q5aeeQNJL1OLp536bqiR/LsUbsrk/AAEZ64fDvG861rEItSpmKhK 33hQ== MIME-Version: 1.0 X-Received: by 10.224.14.132 with SMTP id g4mr10476226qaa.26.1388475337244; Mon, 30 Dec 2013 23:35:37 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.52.8 with HTTP; Mon, 30 Dec 2013 23:35:37 -0800 (PST) Date: Mon, 30 Dec 2013 23:35:37 -0800 X-Google-Sender-Auth: 9i4BXUDaZMCSQdV-m7zYdC9zUVE Message-ID: Subject: ipv6 lock contention with parallel socket io From: Adrian Chadd To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Dec 2013 07:35:38 -0000 Hi, I've noticed a bunch of lock contention occurs when doing highly parallel (> 4096 sockets) TCP IPv6 traffic. The contention is here: root@c001-freebsd11:/home/adrian/git/github/erikarn/libiapp # sysctl debug.lock.prof.stats | head -2 ; sysctl debug.lock.prof.stats | sort -nk4 | tail -10 debug.lock.prof.stats: max wait_max total wait_total count avg wait_avg cnt_hold cnt_lock name 507 1349 116285 233979 617482 0 0 0 11927 /usr/home/adrian/work/freebsd/head/src/sys/netinet/tcp_hostcache.c:291 (sleep mutex:tcp_hc_entry) 499 995 122943 479346 617480 0 0 0 104210 /usr/home/adrian/work/freebsd/head/src/sys/netinet6/in6_src.c:885 (sleep mutex:rtentry) 515 202 493751 581039 617481 0 0 0 12779 /usr/home/adrian/work/freebsd/head/src/sys/netinet6/in6.c:2376 (rw:lle) 1872 2020 1542355 1529313 617481 2 2 0 97308 /usr/home/adrian/work/freebsd/head/src/sys/netinet6/nd6.c:2229 (rw:if_afdata) 494 1066 141964 1892922 617481 0 3 0 503429 /usr/home/adrian/work/freebsd/head/src/sys/net/flowtable.c:1251 (sleep mutex:rtentry) 388 1121 161966 2152377 617482 0 3 0 397770 /usr/home/adrian/work/freebsd/head/src/sys/netinet/tcp_output.c:1198 (sleep mutex:rtentry) 7 849 603349 2431982 499778 1 4 0 279708 /usr/home/adrian/work/freebsd/head/src/sys/kern/subr_turnstile.c:551 (spin mutex:turnstile chain) 539 1171 844350 5776354 1852441 0 3 0 1254017 /usr/home/adrian/work/freebsd/head/src/sys/net/route.c:380 (sleep mutex:rtentry) 203 2849 851312 7862530 617481 1 12 0 139389 /usr/home/adrian/work/freebsd/head/src/sys/netinet6/nd6.c:1894 (rw:if_afdata) 36 2401 363853 18179236 508578 0 35 0 125063 /usr/home/adrian/work/freebsd/head/src/sys/netinet6/ip6_input.c:701 (rw:if_afdata) root@c001-freebsd11:/home/adrian/git/github/erikarn/libiapp # .. it's the IF_ADATA lock surrounding the lla_lookup() calls. Now: * is there any reason this isn't an rmlock? * the instance early on in nd6_output_lle() isn't taking the read lock, it's taking the full lock. Is there any reason for this? I don't have much experience or time to spend on optimising the ipv6 code to scale better but this seems like one of those things that'll bite us in the ass as the amount of ipv6 deployed increases. Does anyone have any ideas/suggestions on how we could improve things? Thanks, -a