From owner-freebsd-bugs@FreeBSD.ORG Mon Aug 11 21:52:51 2014 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50CCB544 for ; Mon, 11 Aug 2014 21:52:51 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 382A629EA for ; Mon, 11 Aug 2014 21:52:51 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s7BLqp6P067935 for ; Mon, 11 Aug 2014 21:52:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 192599] New: zero window and persist timer not set Date: Mon, 11 Aug 2014 21:52:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jlott@averesystems.com X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Aug 2014 21:52:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192599 Bug ID: 192599 Summary: zero window and persist timer not set Product: Base System Version: 9.2-STABLE Hardware: Any OS: Any Status: Needs Triage Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: jlott@averesystems.com Created attachment 145689 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=145689&action=edit Proposed patch We've been seeing a problem where a tcp connection is stuck in a zero window condition and even though the client has opened more window space, our FreeBSD box never sends any more. After some analysis it appears that the FreeBSD box is not sending zero window probes, because the persist timer did not get set (we can see in kgdb that the tcpcb shows 0 window, there is data in the socket buffer, but the persist timer is not active). After looking over the code for a while, I think I see the problem. When tcp_output chooses to send a packet, it never arms the persist timer. This causes a problem in the following scenario: 1. A --> B: packet containing enough data to fill the window 2. B --> A: ACK for #1 + new data (0 window advertisement) 3. A --> B: ACK for #2, 0 len packet In this case, A will not activate the persist timer, because it chose to send a packet. Unless tcp_output is called for some other reason (delayed ack timer, another input packet from B, socket syscall), A will not send zero window probes. I was finally able to recreate this condition by setting an very small window and running programs that send very specific sequences of packets without calling recv (purposefully forcing a zero window condition). Here is a packet capture that shows the sequence: A == 10.2.15.69 == FreeBSD 9.2 B == 10.2.14.61 == FreeBSD 8.2 16:19:49.664790 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [S], seq 2362665163, win 4300, options [mss 1460,nop,wscale 6,sackOK,TS val 88804503 ecr 0], length 0 16:19:49.664821 IP 10.2.15.69.12345 > 10.2.14.61.23133: Flags [S.], seq 3306387947, ack 2362665164, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1605043666 ecr 88804503], length 0 16:19:49.664859 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [.], ack 1, win 67, options [nop,nop,TS val 88804503 ecr 1605043666], length 0 16:19:49.664921 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [P.], seq 1:101, ack 1, win 67, options [nop,nop,TS val 88804503 ecr 1605043666], length 100 16:19:49.665137 IP 10.2.15.69.12345 > 10.2.14.61.23133: Flags [P.], seq 1:3001, ack 101, win 2046, options [nop,nop,TS val 1605043666 ecr 88804503], length 3000 16:19:49.665208 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [P.], seq 101:1321, ack 1449, win 45, options [nop,nop,TS val 88804503 ecr 1605043666], length 1220 16:19:49.666195 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [.], seq 1321:2769, ack 3001, win 21, options [nop,nop,TS val 88804504 ecr 1605043666], length 1448 16:19:49.666205 IP 10.2.15.69.12345 > 10.2.14.61.23133: Flags [.], ack 2769, win 2004, options [nop,nop,TS val 1605043667 ecr 88804503], length 0 16:19:49.666207 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [P.], seq 2769:2771, ack 3001, win 21, options [nop,nop,TS val 88804504 ecr 1605043666], length 2 16:19:49.667183 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [.], seq 2771:4219, ack 3001, win 21, options [nop,nop,TS val 88804505 ecr 1605043667], length 1448 16:19:49.667190 IP 10.2.15.69.12345 > 10.2.14.61.23133: Flags [.], seq 3001:4345, ack 4219, win 1982, options [nop,nop,TS val 1605043668 ecr 88804504], length 1344 16:19:49.667193 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [P.], seq 4219:4221, ack 3001, win 21, options [nop,nop,TS val 88804505 ecr 1605043667], length 2 16:19:49.766487 IP 10.2.14.61.23133 > 10.2.15.69.12345: Flags [P.], seq 4221:4321, ack 4345, win 0, options [nop,nop,TS val 88804605 ecr 1605043668], length 100 16:19:49.766499 IP 10.2.15.69.12345 > 10.2.14.61.23133: Flags [.], ack 4321, win 1980, options [nop,nop,TS val 1605043768 ecr 88804505], length 0 The important packets are the last four: 1. A --> B: length 1344, fills the remaining window 2. B --> A: length 2, does not ack additional data, delayed ack timer is set 3. B --> A: length 100, acks #1, immediate ack (delayed ack timer cancelled, tcp_output called with ACKNOW) 4. A --> B: length 0, acks #1 and #2, because a packet is sent tcp_output does not activate the persist timer. I would normally expect A to begin sending zero-window probes, but (since it didn't activate the persist timer) it does not. Using kgdb, I can see that the persist timer is not set, only the keep timer is set. This is kgdb on "A": (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->snd_nxt $5 = 3306392292 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->snd_max $6 = 3306392292 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->snd_una $7 = 3306392292 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->snd_wnd $8 = 0 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->snd_cwnd $9 = 4380 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->t_timers->tt_rexmt->c_flags $11 = 16 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->t_timers->tt_persist->c_flags $12 = 16 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->t_timers->tt_keep->c_flags $13 = 22 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->t_timers->tt_2msl->c_flags $14 = 16 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->t_timers->tt_delack->c_flags $15 = 16 (kgdb) print ((struct tcpcb*)(0xfffffe02ae289b70))->t_inpcb->inp_socket.so_snd.sb_cc $16 = 1656 There is zero window, data in the socket buffer, and the persist timer is not set. My proposed fix is attached. If you send a 0-length packet, but there is data is the socket buffer, and neither the rexmt or persist timer is already set, then activate the persist timer. -- You are receiving this mail because: You are the assignee for the bug.