From owner-freebsd-current@freebsd.org Tue Sep 11 09:41:34 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4EBA108AAC5 for ; Tue, 11 Sep 2018 09:41:34 +0000 (UTC) (envelope-from zhongcy95@gmail.com) Received: from mail-ua1-x92e.google.com (mail-ua1-x92e.google.com [IPv6:2607:f8b0:4864:20::92e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43ACF8E649 for ; Tue, 11 Sep 2018 09:41:34 +0000 (UTC) (envelope-from zhongcy95@gmail.com) Received: by mail-ua1-x92e.google.com with SMTP id f4-v6so19897065uao.10 for ; Tue, 11 Sep 2018 02:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=kgZPWZOCWFfRMZiPhNF8QxfrDPCQMO4OUlUjFfvHU7Y=; b=hA1dxAW4u7hg0Goreywqp3WSvFprQyTlR6iRD0pk5P5eHiWaU3/fbRncY7GaADsKaI aB4zXqFONuorP5gHArW5IBqhPyKHg2MoTPyAVJuIfYYipGylYBv2naqgfX6xz2rKpUoB 1WIXsthJI5MIX0Nj8Ur9GuL+IqLxAd9dolY1diHfxmYhGkP7STM+VzfOVPvCiEY3EUcW kKXifyMJJo7twg6beHpGSb/QORPNcWocjX4Zs21sPVjD5Hcf5v96hS5+AJFqqlHelmPd jOGIDztpy2TB5CwmDx9uXB/xwL+cTH6xC2AP0BzcHinyw1O5dKtBUVSdvlilwxgESd2h Ts+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=kgZPWZOCWFfRMZiPhNF8QxfrDPCQMO4OUlUjFfvHU7Y=; b=VBmPHYvWx+D4/58JqtssHI+sETMAVbhk9i7VznDlzm/gx7/mWuRA4Zsl49EwNeOqEm O2NTxE5PtP26/w6DTHZXutg8f4ykgC/D2q0kLWbrTfQgc5lSyGTWG1jhCJCa1CdG9OY9 a4WQTXjXtiJGKM/7/wuDcDErZIwhMWJsOJEV9WXzNNq1csP9KFB0VUa3W6RJdMiBopZq WwgDlmHydJD8lTkcPnMk6b3dtxbwti4V/Z4Yy0Lz8+1dJ/zeRzIgKmY4ZxbqffJTLJDc n59udkZyasRJWGmOlGJz/Fgx1UgDC+GdLclXbY+XbKgaU0RCupnobdHUME2jxHuAYK71 i8bA== X-Gm-Message-State: APzg51DkLhlYccD+NgTM670+vOsgDdWq8BLDOBo73cCkyjWW1EzexUQj yN9Pc5zJBSm1ryQoxqcga5F2qlz4Sxo1rbVkEbMRPmoX X-Google-Smtp-Source: ANB0VdZIDxVjGLpTDE5zs2qS/FMqD4UL5rqgaI+kp07GG4w9s80MUX82v55ajRSf1kVsnBWNkJvvRr1ChzUmHa3xDrI= X-Received: by 2002:a67:2442:: with SMTP id k63-v6mr8534710vsk.67.1536658893334; Tue, 11 Sep 2018 02:41:33 -0700 (PDT) MIME-Version: 1.0 From: Chenyang Zhong Date: Tue, 11 Sep 2018 17:41:20 +0800 Message-ID: Subject: TCP RACK performance To: freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Sep 2018 09:41:35 -0000 Hi, I am really excited to see that @rrs from Netflix is adding TCP RACK and High Precision Timer System to the kernel, so I built a kernel (r338543) and ran some test. I used the following kernel config, as suggested in commit rS334804. makeoptions WITH_EXTRA_TCP_STACKS=1 options TCPHPTS After booting the new kernel, I loaded the tcp_rack.ko, # kldload tcp_rack and checked the sysctl to make sure rack is there. # sysctl net.inet.tcp.functions_available net.inet.tcp.functions_available: Stack D Alias PCB count freebsd * freebsd 3 rack rack 0 I ran the first test with the default stack. I was running iperf3 over a wireless network where rtt was fluctuating but no packet loss. Here is a ping result summary. The average and stddev of rtt is relatively high. 39 packets transmitted, 39 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.920/40.206/124.094/39.093 ms Here is the iperf3 result of a 30-second benchmark. [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 328 MBytes 91.8 Mbits/sec 62 sender [ 5] 0.00-30.31 sec 328 MBytes 90.9 Mbits/sec receiver Then I switched to the new RACK stack. # sysctl net.inet.tcp.functions_default=rack net.inet.tcp.functions_default: freebsd -> rack There was a 10% - 15% performance loss after running the same iperf3 benchmark. Also, the number of retransmissions increased dramatically. [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 286 MBytes 79.9 Mbits/sec 271 sender [ 5] 0.00-30.30 sec 286 MBytes 79.0 Mbits/sec receiver I then ran iperf3 on a Linux machine with kernel 4.15, which uses RACK by default. I verified that through sysctl: # sysctl net.ipv4.tcp_recovery net.ipv4.tcp_recovery = 1 The iperf3 result showed the same speed with the default freebsd stack, and the number of retransmission matched the RACK stack on freebsd. [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-30.00 sec 330 MBytes 92.3 Mbits/sec 270 sender [ 4] 0.00-30.00 sec 329 MBytes 92.1 Mbits/sec receiver I am not sure whether the performance issue is related to my configuration or to the new implementation of RACK on FreeBSD. I am glad to offer more information if anyone is interested. Thanks again for all the hard work. I cannot wait to see TCP BBR on FreeBSD. Best, Chenyang