From owner-freebsd-net@FreeBSD.ORG Mon Nov 4 21:25:15 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1C80E857 for ; Mon, 4 Nov 2013 21:25:15 +0000 (UTC) (envelope-from freebsd-net@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D00662408 for ; Mon, 4 Nov 2013 21:25:14 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1VdRdu-0003ji-R7 for freebsd-net@freebsd.org; Mon, 04 Nov 2013 22:25:02 +0100 Received: from 217.30.88.7 ([217.30.88.7]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 04 Nov 2013 22:25:02 +0100 Received: from jcharbon by 217.30.88.7 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 04 Nov 2013 22:25:02 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-net@freebsd.org From: "Julien Charbon" Subject: TCP stack lock contention with short-lived connections Date: Mon, 04 Nov 2013 22:21:04 +0100 Lines: 29 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 217.30.88.7 User-Agent: Opera Mail/1.0 (MacIntel) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Nov 2013 21:25:15 -0000 Hi list, just a follow-up of vBSDCon discussions about FreeBSD TCP performances with short-lived connections. In summary: Using a single NIC receive queue, the TCP stack performance results with short-lived connections on FreeBSD are great: We got 52,000 (52k) TCP queries per second (QPS) before being CPU bound, we achieved the same result on Linux. However, when configuring 4 NIC receive queues each bound to a different core of the same CPU, results are lower than expected: We got 56k QPS, where we reached 200k QPS on Linux on same hardware. Basically, FreeBSD TCP stack does not scale well across multiple receive queues/CPU cores due to a lock contention. I have put technical and how-to-repeat details in below PR: kern/183659: TCP stack lock contention with short-lived connections http://www.freebsd.org/cgi/query-pr.cgi?pr=183659 We are currently working on this performance improvement effort; it will impact only the TCP locking strategy not the TCP stack logic itself. We will share on freebsd-net the patches we made for reviewing and improvement propositions; anyway this change might also require enough eyeballs to avoid tricky race conditions introduction in TCP stack. Thanks. -- Julien