From owner-freebsd-transport@freebsd.org Fri Feb 17 22:37:41 2017 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA1C1CE251E for ; Fri, 17 Feb 2017 22:37:41 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: from mail-wr0-x235.google.com (mail-wr0-x235.google.com [IPv6:2a00:1450:400c:c0c::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3428D1724 for ; Fri, 17 Feb 2017 22:37:41 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: by mail-wr0-x235.google.com with SMTP id 89so32776819wrr.3 for ; Fri, 17 Feb 2017 14:37:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=to:from:subject:message-id:date:user-agent:mime-version; bh=BMk0vuCdRlCUMbYo0JQZkXMqJSHLQKRsURV+df24x5s=; b=WVrF4A2FpgYOF+PRCP5rwnPvOLh0c60EGdYTp0/7LMy3q3wyUEuLL5ig79CjCM9ALe ao/0wIZX3ACchwdWPT6pSBIHix3mRxT1KE2ohVjeyRmkDHX7GIZMb3J30ujsLlNa+2sF dI/zNwJyr8zd9YFAN3k9rR0Yq4qPjgLj5CV3jl6gpoOCf1bWWKytTNoDeseMJEA6l01z lVUnt+40GRJU9p+akllPabuymf5vu0G+SxQwDlN7SvYx144wuag2BxLiTgDfsQ03n+M7 y+BIEcpvAyonqjwzJ+vkqF4BKLyZ4P0AT0/NClZzntLzmD0KpP5uMkUOWm/t/hefMecx RI0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version; bh=BMk0vuCdRlCUMbYo0JQZkXMqJSHLQKRsURV+df24x5s=; b=ChpSa0lTdJPW0XfhT0C6TxPdyp53Dh2p7JxixSZ0zeKxec7LpHko6j4d5dFmOskdAn fyVSLcZQwa+xg38/CKMQ/TZSqR2MlJMzokB0/KPgE2oizfzhyxjy2FFmzaVjfJ95H1v0 m2RFLR1s3Ze9IVdLfx/sWviytS9L5lX8IpW2uD2K3gFaDSI+ViE/qj9qM52H8qzDGNZ1 YGOaiz5ZMyIZfgOjjYARiIgNUneRni3LZcsD+vSLSfcX1q3Mdvxh28Wqn1Hmqo3fdhFX J/UiMs7vkW4KcxqDAf+ZEOVlcpEbBJkC1o2f29Qerd43MgAGwpn3DsaUEznR9BpljGJ0 amPA== X-Gm-Message-State: AMke39msuasUrORP/N+h59KQsh3Jzml6uI7/WZdq7V/+5qnYoI7em9dcCp3s0EkfRhQsTO+B X-Received: by 10.223.131.193 with SMTP id 59mr7730841wre.186.1487371059201; Fri, 17 Feb 2017 14:37:39 -0800 (PST) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id r195sm3280776wme.25.2017.02.17.14.37.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Feb 2017 14:37:38 -0800 (PST) To: freebsd-transport@freebsd.org From: Steven Hartland Subject: TCP Receive buffer scaling without Timestamps Message-ID: Date: Fri, 17 Feb 2017 22:37:44 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2017 22:37:41 -0000 While investigating a recent thread on freebsd-net@ I became aware of the fact that FreeBSD's receive buffer scaling doesn't work when the TCP session fails to negotiate timestamp support. The result of this when downloading content from Amazon S3 and likely other locations Linux is significantly faster without manual tuning. In my tests with a latency of ~17ms on a 1Gbps connection FreeBSD downloaded from AWS S3 at only 3MB/s where the same spec Linux box got 30MB/s. Manually increasing net.inet.tcp.recvspace FreeBSD easily matched Linux performance, but obviously that's clunky, not everyone will do it and hence.... Initial research seems to indicate Linux receive buffer scaling will use timestamps, if they are available, but also falls back to RTT estimates to provide buffer scaling. The benefits of this are clear, so I wanted to kick of thread to highlight this and see if this is something we could address, given its highly likely this is going to be having a marked impact on a significant amount of our users. So the questions: 1. Has anyone looked at this issue before or is working on it? 2. Do people think we could add support for RTT estimates and enable receive scaling without major tcp stack changes? The original thread can be seen here: https://lists.freebsd.org/pipermail/freebsd-net/2017-February/047234.html Regards Steve