From owner-freebsd-net@FreeBSD.ORG Wed Nov 24 02:32:21 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B2AD116A4CE for ; Wed, 24 Nov 2004 02:32:21 +0000 (GMT) Received: from insomnia.benzedrine.cx (insomnia.benzedrine.cx [62.65.145.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id B4CF943D49 for ; Wed, 24 Nov 2004 02:32:20 +0000 (GMT) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (dhartmei@localhost [127.0.0.1]) iAO2WHRh012785 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Wed, 24 Nov 2004 03:32:17 +0100 (MET) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.13.1/8.12.10/Submit) id iAO2WGCR013662; Wed, 24 Nov 2004 03:32:16 +0100 (MET) Date: Wed, 24 Nov 2004 03:32:16 +0100 From: Daniel Hartmeier To: Pawel Worach Message-ID: <20041124023216.GH9247@insomnia.benzedrine.cx> References: <419EBE2E.9080108@telia.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <419EBE2E.9080108@telia.com> User-Agent: Mutt/1.4.1i cc: freebsd-net@freebsd.org Subject: Re: SACK (and PF) wierdness X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Nov 2004 02:32:21 -0000 Pawel kindly supplied a tcpdump of an entire connection. We identified the point where pf drops the packet because it sees a violation of the advertised sequence window. I think we can show that there is some problem with the TCP code, possibly related to SACK. Hence, I'd like to ask anyone familar with TCP and SACK to help. You can find the output on http://www.benzedrine.cx/pawel.txt (328k) This is an active FTP data connection, from FTP server 192.168.1.10:20 to FTP client 192.168.1.200:64828, where payload is only flowing from server to client. Both hosts run recent FreeBSD 6-current. The dump consists of 1571 packets, 940 from server to client and 631 from client to server. pf complains about the 1572th packet (not found in the dump, as it was blocked): Nov 22 23:25:20 darkstar kernel: pf: BAD state: TCP 192.168.1.10:20 192.168.1.10:20 192.168.1.200:64828 [lo=1185380879 high=1185380879 win=33304 modulator=0 wscale=1] [lo=1046638749 high=1046705357 win=33304 modulator=0 wscale=1] 4:4 A seq=1185380879 ack=1046638749 len=1448 ackskew=0 pkts=940:631 dir=out,fwd Nov 22 23:25:20 darkstar kernel: pf: State failure on: 1 | This means the server was trying to send 1448 bytes from th_seq 1185380879, but that violated the upper boundary of what pf thinks the client may accept. pf calculates this upper boundary from th_ack + th_win from the client. Let's look at what the client sent last: 23:25:20.203732 192.168.1.200.64828 > 192.168.1.10.20: . [tcp sum ok] 1046638749:1046638749(0) ack 1185318615 win 31132 (DF) [tos 0x8] (ttl 64, id 54682, len 64) Windows scaling is enabled, and both peers' scaling factors are 2^1, so the upper boundary advertised is th_ack 1185318615 + 2 * th_win 31132 == 1185380879 Why does the server try to send 1448 bytes from 1185380879 upwards? This is a violation of the client's window by the server, right? If you look at some prior packets from the client, from 23:25:20.145089 to 23:25:20.203732, it sent 33 ACKs with equal th_ack 1185314271 but different first SACK confirmations. We've looked at several examples like this, one common thing was that the last packet before the drop was always a zero-payload packet from the server to the client at exactly the client's advertised window border. Maybe that rings a bell for someone. Any ideas welcome :) Daniel