From owner-freebsd-net@FreeBSD.ORG Wed May 20 14:57:49 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 558FECA5; Wed, 20 May 2015 14:57:49 +0000 (UTC) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E4FA1391; Wed, 20 May 2015 14:57:49 +0000 (UTC) Received: by igbpi8 with SMTP id pi8so104469186igb.1; Wed, 20 May 2015 07:57:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hkhd3DBxXDDo0g7jt/t1apapy5ujMOoXXjG8yvGQxhA=; b=C09ctowRUoJFO5C8rmDZXvxoVZHrnl75t7yP9UQh87mbUMdw96nzXwZ8/UXcsAtAtW GFrsgH/+Vrggc+C+JsVJRjsnrSeZeK/NXNvCqnKCO6kv3Wy6pBD7JsFmDVKtU/5oe62A qfKTxaHbrwEQywWV5MfDWQ5u/ioHeDF7J0KBdb3H9gTiB9yAViAIuilYLg5BqvbjD3P2 Vw3cAgB2bK6j6KosClU/P2kPMo9ca1kvYMqWJfSQxU9+PnkYF7hC1coX4x6k2fZlduLj aZeg9zlFGdJ74fE1lsfaAsLNPc/yCanwOvLh/hYFn4I03jC6nkPUY2tcMhu3HX/mqkPU tHXA== MIME-Version: 1.0 X-Received: by 10.43.163.129 with SMTP id mo1mr45588685icc.61.1432133868542; Wed, 20 May 2015 07:57:48 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Wed, 20 May 2015 07:57:48 -0700 (PDT) In-Reply-To: <555C8BC9.2080304@freebsd.org> References: <537F39DF.1090900@verisign.com> <537FB51D.2060401@verisign.com> <537FBFA4.1010902@FreeBSD.org> <53834368.6070103@verisign.com> <555C8BC9.2080304@freebsd.org> Date: Wed, 20 May 2015 07:57:48 -0700 X-Google-Sender-Auth: wI4aHP4MOrfabFMQM4W9U56qnko Message-ID: Subject: Re: TCP stack lock contention with short-lived connections From: Adrian Chadd To: Julien Charbon Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 14:57:49 -0000 On 20 May 2015 at 06:27, Julien Charbon wrote: > > Hi, > > On 26/05/14 15:36, Julien Charbon wrote: >> On 23/05/14 23:37, Navdeep Parhar wrote: >>> On 05/23/14 13:52, Julien Charbon wrote: >>>> On 23/05/14 14:06, Julien Charbon wrote: >>>>> On 27/02/14 11:32, Julien Charbon wrote: >>>>>> On 07/11/13 14:55, Julien Charbon wrote: >>>>>>> On Mon, 04 Nov 2013 22:21:04 +0100, Julien Charbon >>>>>>> wrote: >>>>>>>> I have put technical and how-to-repeat details in below >>>>>>>> PR: >>>>>>>> >>>>>>>> kern/183659: TCP stack lock contention with short-lived >>>>>>>> connections >>>>>>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=183659 >>>>>>>> >>>>>>>> We are currently working on this performance improvement >>>>>>>> effort; it will impact only the TCP locking strategy not >>>>>>>> the TCP stack logic itself. We will share on freebsd-net >>>>>>>> the patches we made for reviewing and improvement >>>>>>>> propositions; anyway this change might also require enough >>>>>>>> eyeballs to avoid tricky race conditions introduction in >>>>>>>> TCP stack. >>>> >>>> [...] >> >> Below, just for your information, more details on context of these >> changes: >> >> o The rough consensus at BSDCan was that there is a shared interest for >> scalability improvement of TCP workloads with potential high rate of >> connections establishment and tear-down. >> >> o Our requirements for this task were: >> - Achieve more than 100k TCP connections per second without dropping a >> single packet in reception >> - Use a strategy that does not require to change all network stacks in >> a row (TCP, UDP, RAW, etc.) >> - Be able to progressively introduce better performance, leveraging >> already in place mutex strategy >> - Keep the TCP stack stable (obviously) > > For people interested about this short-lived TCP connection scalability > effort, you can subscribe to the review of our latest (and biggest so > far) change: > > Decompose TCP INP_INFO lock to increase short-lived connections scalability > https://reviews.freebsd.org/D2599 > > The main goal of this review is ideally to start the rough discussion > before BSDCan (and then discuss details in person at BSDCan), and ease > tests by other people with more exotic configurations (thanks Adrian for > your early tests). This patch still improves the short-lived TCP > connection rate (setup and teardown) from 60k/sec to 150k/sec. Hi! I'm using this in our testing lab at work. I get to around 105k/sec, but I think that's primarily because of other lock contention in the kernel (things doing unnecessary ioctl()s.) -adrian