Date: Mon, 01 Dec 2008 12:19:40 -0300 From: Paulo Fragoso <paulo@nlink.com.br> To: Andre Oppermann <andre@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: FreeBSD 7.1 tcp problem (syncache)? Message-ID: <4934008C.1070303@nlink.com.br> In-Reply-To: <4933F7D6.50805@freebsd.org> References: <4933EC58.5030204@nlink.com.br> <4933F7D6.50805@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------040306050501000500000100
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Em 01/12/2008 11:42, Andre Oppermann escreveu:
> Paulo Fragoso wrote:
>> Hi,
>>
>> We was using one machine with FreeBSD 6.4-RELEASE running
>> apache-worker-2.2.3 + mysql, this server can answer high request from
>> one client using ab:
>>
>> {client}$ ab -n 2000 -c 1000 http://system_using_6.4-RELEASE
>> ...
>> Benchmarking ***** (be patient)
>> Completed 200 requests
>> Completed 400 requests
>> Completed 600 requests
>> Completed 800 requests
>> Completed 1000 requests
>> Completed 1200 requests
>> Completed 1400 requests
>> Completed 1600 requests
>> Completed 1800 requests
>> Finished 2000 requests
>> ...
>> {client}$
>>
>> Using other hardware whit FreeBSD 7.1-PRERELEASE running
>> apache-worker-2.2.9_5 + mysql, we have a poor result:
>>
>> {client}$ ab -n 2000 -c 1000 http://system_using_7.1-PRERELEASE
>> ...
>> Test aborted after 10 failures
>>
>> apr_connect(): Invalid argument (22)
>> {client}$
>>
>> Looking for a problem on new server log we found:
>>
>> kernel: TCP: [client]:50197 to [server]:80 tcpflags 0x4<RST>;
>> syncache_chkrst: Spurious RST without matching syncache entry
>> (possibly syncookie only), segment ignored
>> kernel: TCP: [client]:53845 to [server]:80 tcpflags 0x4<RST>;
>> syncache_chkrst: Spurious RST without matching syncache entry
>> (possibly syncookie only), segment ignored
>> kernel: TCP: [client]:53845 to [server]:80 tcpflags 0x4<RST>;
>> syncache_chkrst: Spurious RST without matching syncache entry
>> (possibly syncookie only), segment ignored
>
> This error is harmless and occurs when the client closes the socket
> before the TCP session has terminated. The RST have crossed and hit
> the syncache. The same happens on 6.4 but it never told you about it.
Ok, but we can not understand why same solution fails using FreeBSD 7.1
and works fina with
>
>> All sysctl and apache conf are same on both server, is there a tcp
>> problem with FreeBSD 7.x?
>
> Are you using the HTTP accept filter on the server? This may cause
> the listen accept queue to overflow.
No.
server-7.1# kldstat
Id Refs Address Size Name
1 4 0xffffffff80100000 b302e0 kernel
2 1 0xffffffffa3391000 8eba ipfw.ko
3 1 0xffffffffa33fe000 1ea green_saver.ko
server-7.1#
server-6.4# kldstat
Id Refs Address Size Name
1 4 0xffffffff80100000 a380f0 kernel
2 1 0xffffffff979f0000 8566 ipfw.ko
3 1 0xffffffff97a35000 1dd green_saver.ko
server-6.4#
>
>
> Please post the output of "netstat -n -s -p tcp" to further diagnose
> the problem.
netstat output from two servers was attached.
Our tests are based on 08 ab tests, changing -n and -c parameters at
client machine, starting with:
ab -n 100 -c 10 http://server/path_to_cgi
and finishing with:
ab -n 5000 -c 1000 http://server/path_to_cgi
The old server using FreeBSD-6.4 and running on worst hardware can
answer all request with some errors:
FreeBSD 6.4:
ab -n -c
------------------------------------------------
100 10
1000 10 (8 errors)
1000 50 (12 errors)
1000 100 (12 errors)
1000 500 (11 errors)
2000 500 (24 errors)
2000 1000 (24 errors)
5000 1000 (4997 errors)
but we can not receive any answer for two last test using server-7.1
FreeBSD 7.1:
ab -n -c
------------------------------------------------
100 10 (1 error)
1000 10 (5 errors)
1000 50 (7 errors)
1000 100 (8 errors)
1000 500 (11 errors)
2000 500 (15 errors)
2000 1000 no answer
5000 1000 no answer
ab return this error on test machine:
--------------------------------------
Test aborted after 10 failures
apr_connect(): Invalid argument (22)
--------------------------------------
Is there a new limit in FreeBSD-7.1?
Paulo Fragoso
--------------040306050501000500000100
Content-Type: text/plain;
name="netstat-7.1.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="netstat-7.1.txt"
server-7.1# netstat -n -s -p tcp
tcp:
75063 packets sent
31728 data packets (27702356 bytes)
2 data packets (1864 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
34657 ack-only packets (9002 delayed)
0 URG only packets
0 window probe packets
142 window update packets
8532 control packets
80919 packets received
28917 acks (for 15918245 bytes)
9425 duplicate acks
0 acks for unsent data
26168 packets (3681069 bytes) received in-sequence
893 completely duplicate packets (76110 bytes)
0 old duplicate packets
15 packets with some dup. data (1770 bytes duped)
60 out-of-order packets (86880 bytes)
10 packets (0 bytes) of data after window
0 window probes
3323 window update packets
2 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
3 connection requests
11907 connection accepts
0 bad connection attempts
618 listen queue overflows
6398 ignored RSTs in the windows
11910 connections established (including accepts)
9740 connections closed (including 3266 drops)
24 connections updated cached RTT on close
24 connections updated cached RTT variance on close
0 connections updated cached ssthresh on close
0 embryonic connections dropped
28917 segments updated rtt (of 17512 attempts)
3 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
281 correct ACK header predictions
13610 correct data packet header predictions
12526 syncache entries added
6 retransmitted
8 dupsyn
0 dropped
11907 completed
28 bucket overflow
0 cache overflow
10 reset
0 stale
618 aborted
0 badack
0 unreach
0 zone failures
12526 cookies sent
37 cookies received
1 SACK recovery episode
1 segment rexmit in SACK recovery episodes
416 byte rexmits in SACK recovery episodes
4 SACK options (SACK blocks) received
31 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
--------------040306050501000500000100
Content-Type: text/plain;
name="netstat-6.4.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="netstat-6.4.txt"
server-6.4# netstat -n -s -p tcp
tcp:
419071 packets sent
357030 data packets (378965034 bytes)
26 data packets (3665 bytes) retransmitted
15 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
44767 ack-only packets (13440 delayed)
0 URG only packets
0 window probe packets
11 window update packets
17237 control packets
315358 packets received
242513 acks (for 373590012 bytes)
12703 duplicate acks
0 acks for unsent data
59283 packets (4705080 bytes) received in-sequence
7 completely duplicate packets (384 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
16 out-of-order packets (3904 bytes)
0 packets (0 bytes) of data after window
0 window probes
11056 window update packets
0 packets received after close
1 discarded for bad checksum
0 discarded for bad header offset fields
0 discarded because packet too short
4537 connection requests
13438 connection accepts
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
15708 connections established (including accepts)
15610 connections closed (including 2579 drops)
4556 connections updated cached RTT on close
4556 connections updated cached RTT variance on close
2 connections updated cached ssthresh on close
2265 embryonic connections dropped
241277 segments updated rtt (of 138901 attempts)
21 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
14 keepalive timeouts
14 keepalive probes sent
0 connections dropped by keepalive
31855 correct ACK header predictions
7454 correct data packet header predictions
13438 syncache entries added
17 retransmitted
0 dupsyn
0 dropped
13438 completed
34 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
13438 cookies sent
34 cookies received
7 SACK recovery episodes
7 segment rexmits in SACK recovery episodes
336 byte rexmits in SACK recovery episodes
119 SACK options (SACK blocks) received
8 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
--------------040306050501000500000100--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4934008C.1070303>