Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 30 Mar 2012 13:49:07 +0200
From:      Andre Oppermann <andre@freebsd.org>
To:        Sergey Smitienko <hunter@comsys.com.ua>
Cc:        freebsd-net@freebsd.org
Subject:   Re: FreeBSD 9.0 generates incorrect SEC/ACK numbers under load
Message-ID:  <4F759DB3.2060706@freebsd.org>
In-Reply-To: <4F7463CF.8010206@comsys.com.ua>
References:  <4F7463CF.8010206@comsys.com.ua>

next in thread | previous in thread | raw e-mail | index | archive | help
On 29.03.2012 15:29, Sergey Smitienko wrote:
> Hello.
>
> I've run into a problem with a web server runing FreeBSD 9.0/amd64. What
> I believe is happening, is what server loses track of correct SEQ/ACK
> numbers
> on some connections. Here is an example:
>
> 15:20:00.347514 IP (tos 0x68, ttl 123, id 1181, offset 0, flags [DF],
> proto TCP (6), length 52)
>      93.72.14.220.49239>  193.178.147.113.80: Flags [S], cksum 0x6995
> (correct), seq 3881466934, win 8192, options [mss 1460,nop,wscale
> 2,nop,nop,sackOK], length 0
> 15:20:00.347526 IP (tos 0x10, ttl 254, id 28065, offset 0, flags [DF],
> proto TCP (6), length 44)
>      193.178.147.113.80>  93.72.14.220.49239: Flags [S.], cksum 0x79fa
> (correct), seq 2151790680, ack 3881466935, win 0, options [mss 1460],
> length 0

The window reported in the SYN/ACK is zero and no window scaling is
reported.  Do you see this on good connections too?

> 15:20:00.361812 IP (tos 0x68, ttl 123, id 1183, offset 0, flags [DF],
> proto TCP (6), length 40)
>      93.72.14.220.49239>  193.178.147.113.80: Flags [.], cksum 0x96c6
> (correct), seq 3881466935, ack 2151790681, win 64240, length 0
> 15:20:00.361869 IP (tos 0x10, ttl 254, id 31305, offset 0, flags [DF],
> proto TCP (6), length 40)
>      193.178.147.113.80>  93.72.14.220.49239: Flags [.], cksum 0x71b7
> (correct), seq 2151790681, ack 3881466935, win 8192, length 0

This packet is normally not necessary and it shouldn't be generated.
Here it opens the window from zero to 8K (which is very small). Do you
see it being generated on good connections too?

> Client sends "GET"  request
> 15:20:48.236181 IP (tos 0x68, ttl 123, id 1353, offset 0, flags [DF],
> proto TCP (6), length 626)
>      93.72.14.220.49239>  193.178.147.113.80: Flags [P.], cksum 0x7fc9
> (correct), seq 3881466935:3881467521, ack 2151790681, win 64240, length 586
>
> and then the "ping-pong" starts:
>
> 15:20:48.236198 IP (tos 0x0, ttl 254, id 63530, offset 0, flags [DF],
> proto TCP (6), length 40)
>      193.178.147.113.80>  93.72.14.220.49239: Flags [.], cksum 0x8a97
> (correct), seq 2991748588, ack 1985077892, win 8760, length 0
> 15:20:48.255998 IP (tos 0x68, ttl 123, id 1357, offset 0, flags [DF],
> proto TCP (6), length 40)
>      93.72.14.220.49239>  193.178.147.113.80: Flags [.], cksum 0x947c
> (correct), seq 3881467521, ack 2151790681, win 64240, length 0
> 15:20:48.256015 IP (tos 0x0, ttl 254, id 53518, offset 0, flags [DF],
> proto TCP (6), length 40)
>      193.178.147.113.80>  93.72.14.220.49239: Flags [.], cksum 0x8a97
> (correct), seq 2991748588, ack 1985077892, win 8760, length 0
> 15:20:48.276084 IP (tos 0x68, ttl 123, id 1360, offset 0, flags [DF],
> proto TCP (6), length 40)
>      93.72.14.220.49239>  193.178.147.113.80: Flags [.], cksum 0x947c
> (correct), seq 3881467521, ack 2151790681, win 64240, length 0
> 15:20:48.276099 IP (tos 0x0, ttl 254, id 42983, offset 0, flags [DF],
> proto TCP (6), length 40)
>      193.178.147.113.80>  93.72.14.220.49239: Flags [.], cksum 0x8a97
> (correct), seq 2991748588, ack 1985077892, win 8760, length 0
> 15:20:48.290914 IP (tos 0x68, ttl 123, id 1361, offset 0, flags [DF],
> proto TCP (6), length 40)
>      93.72.14.220.49239>  193.178.147.113.80: Flags [.], cksum 0x947c
> (correct), seq 3881467521, ack 2151790681, win 64240, length 0



> This happens on about 0.01% of connections. This tcpdump is recorded on
> the 193.178.147.113, before traffic hits the wire.
> So it's not a NIC fault. Server is running nginx and serving static
> content 200-500 request  per second.
>
> Any ideas ?

No smoking gun yet.  The SYN/ACK is very dubious though in the first
place.  For the offset of SEQ and ACK it seems like some sort of inpcb
mixup or 2MSL inference.

What does "sysctl net.inet.tcp" on your server report?

-- 
Andre



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F759DB3.2060706>