Date: Mon, 6 Oct 1997 16:26:28 -0700 (PDT) From: Matthew Dillon <dillon@flash.noc.best.net> To: FreeBSD-gnats-submit@FreeBSD.ORG Subject: kern/4712: TCP stack bug Message-ID: <199710062326.QAA00389@flash.noc.best.net> Resent-Message-ID: <199710062330.QAA07979@hub.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 4712
>Category: kern
>Synopsis: After setting larger tcp buffer sizes with sysctl, tcp protocol connection lockup can occur
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Oct 6 16:30:00 PDT 1997
>Last-Modified:
>Originator: Matthew Dillon
>Organization:
Best Internet Communications
>Release: FreeBSD 2.2-STABLE i386
>Environment:
>Description:
Included below are two tcpdump's. The first tcpdump is a
'telnet www.erols.com 80' and 'GET /' prior to any sysctl's. The second
tcpdump is a 'telnet www.erols.com 80' and 'GET /' after the specified
sysctl's. Note that the lockup involves a delay that erols happened to be
generating between the SYN and SYN-ACK.
I have been able to reproduce the same bug with other WWW sites as well as
with other local FreeBSD machines. With other FreeBSD machines, rather
then locking up, a long delay between the first block of data and
subsequent blocks of data occurs where the machine making the request
waits for a prove from the www server. The remote www server under FreeBSD
sends a window probe with at least 1 byte, which seems to unstick the
connection. Remote NON-FreeBSD boxes often send window probes with 0
bytes, which does not unstick the connection.
See below for additional comments.
16:10:46.196059 205.149.163.43.1031 > 205.252.116.184.80: S 84382581:84382581(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
16:10:46.342111 205.252.116.184.80 > 205.149.163.43.1031: S 2082101445:2082101445(0) ack 84382582 win 31744 <mss 1460>
16:10:46.342225 205.149.163.43.1031 > 205.252.116.184.80: . ack 1 win 17520 (DF) [tos 0x10]
16:10:48.042558 205.149.163.43.1031 > 205.252.116.184.80: P 1:8(7) ack 1 win 17520 (DF) [tos 0x10]
16:10:48.204531 205.252.116.184.80 > 205.149.163.43.1031: . 1:1461(1460) ack 8 win 31744 (DF) [tos 0x10]
16:10:48.205971 205.252.116.184.80 > 205.149.163.43.1031: . 1461:2921(1460) ack 8 win 31744 (DF) [tos 0x10]
16:10:48.207793 205.149.163.43.1031 > 205.252.116.184.80: . ack 2921 win 17520 (DF) [tos 0x10]
16:10:48.312987 205.252.116.184.80 > 205.149.163.43.1031: . 4381:5841(1460) ack 8 win 31744 [tos 0x10]
16:10:48.313094 205.149.163.43.1031 > 205.252.116.184.80: . ack 2921 win 17520 (DF) [tos 0x10]
16:10:48.314425 205.252.116.184.80 > 205.149.163.43.1031: . 5841:7301(1460) ack 8 win 31744 [tos 0x10]
16:10:48.314505 205.149.163.43.1031 > 205.252.116.184.80: . ack 2921 win 17520 (DF) [tos 0x10]
16:10:48.324025 205.252.116.184.80 > 205.149.163.43.1031: . 2921:4381(1460) ack 8 win 31744 [tos 0x10]
... continues to completion
sysctl -w net.inet.tcp.sendspace=65536
sysctl -w net.inet.tcp.recvspace=65536
sysctl -w net.inet.tcp.keepidle=3600
sysctl -w net.inet.tcp.always_keepalive=1
route -n change default -sendpipe 65536 -recvpipe 65536
(NOTE: make sure route to telnet destination has the new sendpipe/recvpipe
values correct. FreeBSD sometimes doesn't pick up the change)
Note that the screwup appears to occur when www.erols.com delays sending
it's ack. This causes the FREEBSD box to send an initial small WINDOW
(164 bytes!!) in packet #3. The remote box sends the 164 bytes, and the
freebsd box then gets locked up and always replies with a 0 window, even
though the 164 bytes were read by telnet and displayed in the window.
16:14:58.112451 205.149.163.43.1043 > 205.252.116.217.80: S 133560210:133560210(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp[|tcp]> (DF) [tos 0x10]
16:14:58.240605 205.252.116.217.80 > 205.149.163.43.1043: S 704725219:704725219(0) ack 133560211 win 31744 <mss 1460>
16:14:58.240714 205.149.163.43.1043 > 205.252.116.217.80: . ack 1 win 164 (DF) [tos 0x10]
16:14:59.241901 205.149.163.43.1043 > 205.252.116.217.80: P 1:8(7) ack 1 win 164 (DF) [tos 0x10]
16:15:00.470076 205.149.163.43.1043 > 205.252.116.217.80: P 1:8(7) ack 1 win 164 (DF) [tos 0x10]
16:15:02.356504 205.252.116.217.80 > 205.149.163.43.1043: . 1:165(164) ack 8 win 31744 (DF) [tos 0x10]
16:15:02.370079 205.149.163.43.1043 > 205.252.116.217.80: . ack 165 win 0 (DF) [tos 0x10]
16:15:08.498257 205.252.116.217.80 > 205.149.163.43.1043: . ack 8 win 31744 [tos 0x10]
16:15:08.498341 205.149.163.43.1043 > 205.252.116.217.80: . ack 165 win 0 (DF) [tos 0x10]
16:15:20.625433 205.252.116.217.80 > 205.149.163.43.1043: . ack 8 win 31744 [tos 0x10]
16:15:20.625544 205.149.163.43.1043 > 205.252.116.217.80: . ack 165 win 0 (DF) [tos 0x10]
.... retries forever, connection is hung ...
>How-To-Repeat:
>Fix:
I believe the bug is in netinet/tcp_output.c somewhere, possibly
where it zero's the window when checking for silly-window syndrome:
if (win < (long)(so->so_rcv.sb_hiwat / 4) && win < (long)tp->t_maxseg)
win = 0;
I have NOT attempted to hack the code or anything, so I do not
know if this is where the bug is. It is just a guess.
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199710062326.QAA00389>