Date: Mon, 04 Jun 2012 10:22:32 +1000 From: Lawrence Stewart <lstewart@freebsd.org> To: Kevin Oberman <kob6558@gmail.com> Cc: freebsd-net@freebsd.org, Andrew Gallatin <gallatin@cs.duke.edu>, Andrew Gallatin <gallatin@myri.com> Subject: Re: Major performance hit with ToS setting Message-ID: <4FCBFFC8.8000402@freebsd.org> In-Reply-To: <CAN6yY1v08qk2VhXFg0Qiz-pMM6md2c_E_kEvA-oqbxuvSN1JDg@mail.gmail.com> References: <CAN6yY1sLxFJ18ANO7nQqLetnJiT-K6pHC-X3yT1dWuWGa0VLUg@mail.gmail.com> <4FBF88CE.20209@cs.duke.edu> <CAN6yY1v%2Bvf=SW7WDGHxCkJtOdj8K3f450jNxFWK_Jc%2B-pFg0nA@mail.gmail.com> <4FC82D6C.4050309@freebsd.org> <CAN6yY1v08qk2VhXFg0Qiz-pMM6md2c_E_kEvA-oqbxuvSN1JDg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 06/03/12 15:18, Kevin Oberman wrote:
> On Fri, Jun 1, 2012 at 2:48 AM, Lawrence Stewart<lstewart@freebsd.org> wrote:
>> On 05/31/12 13:33, Kevin Oberman wrote:
>> [snip]
>>>
>>> I used SIFTR at the suggestion of Lawrence Stewart who headed the
>>>
>>> project to bring plugable congestion algorithms to FreeBSD and found
>>> really odd congestion behavior. First, I do see a triple ACK, but the
>>> congestion window suddenly drops from 73K to 8K. If I understand
>>> CUBIC, it should half the congestion window, not what is happening..
>>> It then increases slowly (in slow start) to 82K. while the slow-start
>>> bytes are INCREASING, the congestion window again goes to 8K while the
>>> SS size moves from 36K up to 52K. It just continues to bound wildly
>>> between 8K (always the low point) and between 64k and 82K. The swings
>>> start at 83K and, over the first few seconds the peaks drop to about
>>> 64K.
>>
>>
>> Oh, and a comment about this behaviour. Dropping back to 8k (1MSS) is only
>> nasty if the TF_{CONG|FAST}RECOVERY flags are *not* set i.e. if you see cwnd
>> grow, drop to 8k with those flags set, and then when the flags are unset,
>> cwnd starts at the value of ssthresh, then that is perfectly normal recovery
>> behaviour. What *is* nasty is if an RTO fires, which will reset cwnd to 8k,
>> ssthresh to 2*MSS and make the connection effectively start from scratch
>> again.
>>
>> There is evidence of RTOs in your siftr output, which is bad news e.g here's
>> one example of 2 side-by-side log lines from your trace:
>>
>> # Direction,time,ssthresh,cwnd,flags
>> i,1338319593.574706,27044,27044,1630544864
>> o,1338319593.831482,16384,8192,1092625377
>>
>> Note the 300ms gap, and how cwnd resets to 1MSS and flags go from 1630544864
>> (TF_WASCRECOVERY|TF_CONGRECOVERY|TF_WASFRECOVERY|TF_FASTRECOVERY) to
>> 1092625377 (TF_WASCRECOVERY|TF_WASFRECOVERY).
>
> What can I say but that you are right. When I looked at the interface
> stats I found that the link overflow drops were through the roof! This
> confuses me a bit since the traffic is outbound and I woudl assume
> from the description on hte Myricom web page that these are input
> drops. A problem a problem with that card? On systems that are
> working "normally", I still see a sharp drop with the ToS bits set,
> but nothing nearly as drastic. Now it is a drop from 4.5G to 728M on a
> cross-country (US) circuit.
>
> I am now looking for issues on the route that might explain the
> performance, but the question of why the drop-of only shows up in
> FreeBSD 8 means something odd is still going on. It is even possible
> that the problem is with 7 and the losses are due to the policy for
> ToS 32 on the path. ToS 32 is less than best effort in our network.
> Maybe the marking was getting lost on 7. Not likely, but possible.
The receiver is FreeBSD 7? If so, have you tuned your reassembly queue
on that machine? If not, that could explain the RTOs you're seeing. Send
through the output of "sysctl net.inet.tcp.reass" and "netstat -sp tcp"
obtained from the receiver immediately before and after running a short
ToS=32 test.
Cheers,
Lawrence
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FCBFFC8.8000402>