Date: Mon, 20 Feb 2017 19:36:47 +1100 From: Lawrence Stewart <lstewart@freebsd.org> To: Steven Hartland <steven.hartland@multiplay.co.uk>, freebsd-transport@freebsd.org Subject: Re: TCP Receive buffer scaling without Timestamps Message-ID: <9d615fdb-c507-b6ef-2abe-e464a7849748@freebsd.org> In-Reply-To: <2cc38c5d-78a9-4ce1-902e-70f2e55f14a2@multiplay.co.uk> References: <b9906198-a026-1403-73e6-279066b4e197@multiplay.co.uk> <be22c783-26d8-321d-d7b1-1edeb27d11e8@freebsd.org> <2cc38c5d-78a9-4ce1-902e-70f2e55f14a2@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On 19/02/2017 12:37, Steven Hartland wrote:
> On 18/02/2017 00:33, Lawrence Stewart wrote:
>> On 18/02/2017 09:37, Steven Hartland wrote:
>> [snip]
>>> So the questions:
>>>
>>> 1. Has anyone looked at this issue before or is working on it?
>>> 2. Do people think we could add support for RTT estimates and enable
>>> receive scaling without major tcp stack changes?
>> We already have a non-TS-enabled RTT estimator in place c.f. the
>> relevant code in tcp_input.c:
>>
>> if ((to.to_flags & TOF_TS) != 0 &&
>> to.to_tsecr) {
>> uint32_t t;
>>
>> t = tcp_ts_getticks() - to.to_tsecr;
>> if (!tp->t_rttlow || tp->t_rttlow > t)
>> tp->t_rttlow = t;
>> tcp_xmit_timer(tp,
>> TCP_TS_TO_TICKS(t) + 1);
>> } else if (tp->t_rtttime &&
>> SEQ_GT(th->th_ack, tp->t_rtseq)) {
>> if (!tp->t_rttlow ||
>> tp->t_rttlow > ticks - tp->t_rtttime)
>> tp->t_rttlow = ticks - tp->t_rtttime;
>> tcp_xmit_timer(tp,
>> ticks - tp->t_rtttime);
>> }
>>
>>
>> The autoscaling implementation just needs to have its insistence on
>> using timestamps stuffed into a cannon and shot in the direction of the
>> sun. This is the relevant block of rcvbuf code from tcp_input.c:
>>
>>
>> if (V_tcp_do_autorcvbuf &&
>> (to.to_flags & TOF_TS) &&
>> to.to_tsecr &&
>> (so->so_rcv.sb_flags & SB_AUTOSIZE)) {
>> if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) &&
>> to.to_tsecr - tp->rfbuf_ts < hz) {
>> if (tp->rfbuf_cnt >
>> (so->so_rcv.sb_hiwat / 8 * 7) &&
>> so->so_rcv.sb_hiwat <
>> V_tcp_autorcvbuf_max) {
>> newsize =
>> min(so->so_rcv.sb_hiwat +
>> V_tcp_autorcvbuf_inc,
>> V_tcp_autorcvbuf_max);
>> }
>> /* Start over with next RTT. */
>> tp->rfbuf_ts = 0;
>> tp->rfbuf_cnt = 0;
>> } else
>> tp->rfbuf_cnt += tlen; /* add up */
>> }
>>
>> It's pretty trivial to fix by anyone so inclined.
> Thanks for the pointers Lawrence, I'm not very familiar with network
> stack but I've had a stab at this, after reading through the current
> code in the blocks you pointed out, along side some experimentation ;-)
Nice, thanks for running with it.
> The results seem pretty encouraging, with S3 downloads with a ~17ms
> latency on a 1Gbps jumping from ~3MB/s to ~80MB/s.
Amazing what a bit of tuning can do :)
> I'd appreciate feedback on the review which can be found here:
> https://reviews.freebsd.org/D9668
ACK, will follow up there.
Cheers,
Lawrence
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9d615fdb-c507-b6ef-2abe-e464a7849748>