From owner-freebsd-transport@freebsd.org Sun Feb 19 01:37:31 2017 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF60CCDB538 for ; Sun, 19 Feb 2017 01:37:31 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: from mail-wr0-x22f.google.com (mail-wr0-x22f.google.com [IPv6:2a00:1450:400c:c0c::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CA631E18 for ; Sun, 19 Feb 2017 01:37:31 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: by mail-wr0-x22f.google.com with SMTP id 35so7149988wrw.0 for ; Sat, 18 Feb 2017 17:37:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=o7Dm83Db0pmZMpUxtYposSFMwrdtB8P79q4xLwNzblI=; b=WxVsMdFkiGWCfqsfAzydwzcbnfDvjppTpW6eLa4lagK4UbiSIRnbNfBMbavSKynyc8 M6uIf7dj1oWGFmAMiTZbSki65MKsSp8soymarHhkkEERy2P2+p3TaRwIyAdttoEEqs8a 56NlQ8JG7+f98f1pxduT1N5SqxDO/jA8zuCygs+4HpcnBuTu796DRnO728T0vPgPSxL2 8yO5RPSWnPZQhgROofEDFIUBYskPBsE0ZaCQ6cCJ2EOPQ2lDz7S+ItNleUHzg0IStZKu F/AZ/+X+vxFIfNeh/jVCWsukntDli36a5C+kgSzXLcnUqjCB8cBhjzjGuL3ukacCi/MJ JknQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=o7Dm83Db0pmZMpUxtYposSFMwrdtB8P79q4xLwNzblI=; b=OnrBSM4gcppY2x37JzCtrNaOejS7tPMUCxgzVvKwy5rHB2qfipBrcczFaXtdbiHk/V Zz5mmuNZukAPyaGUpCOVanFJstW5mysGWXDluGP0KSLdmsSQy43S5/lod0hqrQPRRA5a 9aWLHq0WAgimcVWx+KuTkZ1FNsM++1VpnzHjcxEVhwFuocrOqmggziHv1AFcHUjhiyxY Nz+0nyLLY2CpOXTf/ZzGFV9/9ql0UngkOdFi42v+yG7mCfXGN99ueR9Ux6MIcFwH41kr 3gySoHlYh13NJ7Vdq5kiCW/kcIRejs98T6HcYCIpJFrzJQwAYwagx7IkovQopx5TlPnk tKsQ== X-Gm-Message-State: AMke39kNNP0vDSrtg9ctGTf9S8MDtkKsf1aWShsFyL+5nD77NvoHfKqQbBymw1gNNWakIVfV X-Received: by 10.223.139.213 with SMTP id w21mr11443632wra.108.1487468249290; Sat, 18 Feb 2017 17:37:29 -0800 (PST) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id i73sm7655901wmd.11.2017.02.18.17.37.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 18 Feb 2017 17:37:28 -0800 (PST) Subject: Re: TCP Receive buffer scaling without Timestamps To: Lawrence Stewart , freebsd-transport@freebsd.org References: From: Steven Hartland Message-ID: <2cc38c5d-78a9-4ce1-902e-70f2e55f14a2@multiplay.co.uk> Date: Sun, 19 Feb 2017 01:37:35 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Feb 2017 01:37:31 -0000 On 18/02/2017 00:33, Lawrence Stewart wrote: > On 18/02/2017 09:37, Steven Hartland wrote: > [snip] >> So the questions: >> >> 1. Has anyone looked at this issue before or is working on it? >> 2. Do people think we could add support for RTT estimates and enable >> receive scaling without major tcp stack changes? > We already have a non-TS-enabled RTT estimator in place c.f. the > relevant code in tcp_input.c: > > if ((to.to_flags & TOF_TS) != 0 && > to.to_tsecr) { > uint32_t t; > > t = tcp_ts_getticks() - to.to_tsecr; > if (!tp->t_rttlow || tp->t_rttlow > t) > tp->t_rttlow = t; > tcp_xmit_timer(tp, > TCP_TS_TO_TICKS(t) + 1); > } else if (tp->t_rtttime && > SEQ_GT(th->th_ack, tp->t_rtseq)) { > if (!tp->t_rttlow || > tp->t_rttlow > ticks - tp->t_rtttime) > tp->t_rttlow = ticks - tp->t_rtttime; > tcp_xmit_timer(tp, > ticks - tp->t_rtttime); > } > > > The autoscaling implementation just needs to have its insistence on > using timestamps stuffed into a cannon and shot in the direction of the > sun. This is the relevant block of rcvbuf code from tcp_input.c: > > > if (V_tcp_do_autorcvbuf && > (to.to_flags & TOF_TS) && > to.to_tsecr && > (so->so_rcv.sb_flags & SB_AUTOSIZE)) { > if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) && > to.to_tsecr - tp->rfbuf_ts < hz) { > if (tp->rfbuf_cnt > > (so->so_rcv.sb_hiwat / 8 * 7) && > so->so_rcv.sb_hiwat < > V_tcp_autorcvbuf_max) { > newsize = > min(so->so_rcv.sb_hiwat + > V_tcp_autorcvbuf_inc, > V_tcp_autorcvbuf_max); > } > /* Start over with next RTT. */ > tp->rfbuf_ts = 0; > tp->rfbuf_cnt = 0; > } else > tp->rfbuf_cnt += tlen; /* add up */ > } > > It's pretty trivial to fix by anyone so inclined. Thanks for the pointers Lawrence, I'm not very familiar with network stack but I've had a stab at this, after reading through the current code in the blocks you pointed out, along side some experimentation ;-) The results seem pretty encouraging, with S3 downloads with a ~17ms latency on a 1Gbps jumping from ~3MB/s to ~80MB/s. I'd appreciate feedback on the review which can be found here: https://reviews.freebsd.org/D9668 Its a bit rough ATM in order to facilitate monitoring the behaviour with dtrace. My initial attempt did try to replicate the tests done to facilitate the calls to tcp_xmit_timer, but they couldn't be reused due to the changes the tcp_xmit_timer calls themselves trigger, so I based it off the already calculated and stored smoothed RTT instead. Regards Steve