From owner-freebsd-net@FreeBSD.ORG Thu Oct 17 22:04:57 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id ED3E3FA0 for ; Thu, 17 Oct 2013 22:04:56 +0000 (UTC) (envelope-from bounces+73574-866e-freebsd-net=freebsd.org@sendgrid.me) Received: from o3.shared.sendgrid.net (o3.shared.sendgrid.net [208.117.48.85]) by mx1.freebsd.org (Postfix) with SMTP id 9D2EA2656 for ; Thu, 17 Oct 2013 22:04:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h=from:mime-version:to:subject:content-type:content-transfer-encoding; s=smtpapi; bh=mYF6fODW38ZBgOOsqOBRTrqKWfU=; b=BeynJHkf3/Rl1IffSH +432M7iufT/an1c5r6f+S1alQpt5stJ6MWhtcm5TL3UXlmROmAyXLP6JDYY5NqPJ XyWtt1Tyhmv1PromK+za3lh6N/qw+j2yC3O3zNOOt8rsopJdHgePV96CgNuH0faL LMtUcf+wGMuQuYurXJs/qOIPc= Received: by filter-135.sjc1.sendgrid.net with SMTP id filter-135.12838.52605F071 Thu, 17 Oct 2013 22:04:55 +0000 (GMT) Received: from mail.tarsnap.com (unknown [10.60.208.13]) by mi16 (SG) with ESMTP id 141c87332c5.63c8.87f9c7 for ; Thu, 17 Oct 2013 22:04:54 +0000 (UTC) Received: (qmail 40126 invoked from network); 17 Oct 2013 22:04:53 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by ec2-107-20-205-189.compute-1.amazonaws.com with ESMTP; 17 Oct 2013 22:04:53 -0000 Received: (qmail 23340 invoked from network); 17 Oct 2013 22:03:53 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by clamshell.daemonology.net with SMTP; 17 Oct 2013 22:03:53 -0000 Message-ID: <52605EC9.6090406@freebsd.org> Date: Thu, 17 Oct 2013 15:03:53 -0700 From: Colin Percival User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: LRO causing stretch ACK violations interacts badly with delayed ACKing X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-SG-EID: ISGKkmHlRE12gnWy0TjFyGQSFR1WnpjQdICm3qu6YxEvsV1hzqsglIY99Quqt84nrQERKmjdwWbqOTMhFTL5avN5LcQI9w5KDHvjrpnngvyesP8t3J6pda1eclzrIACA+fwGdpRaKeA5cLTckWl6JAwRMRa1X+FnXW9JVeUPJfo= X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 22:04:57 -0000 Hi all, I know {TSO, LRO, ACKing policy} has been discussed here recently, and I don't want to rehash everything, but I'm seeing some very bad misbehaviour with LRO and delayed ACKing turned on. Running 'fetch -o /dev/null https://www.amazon.com/' on an EC2 instance running FreeBSD 10.0-BETA1: > 00:00:00.000000 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [S], seq 3375534763, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1606713 ecr 0], length 0 > 00:00:00.000754 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [S.], seq 3035209700, ack 3375534764, win 8190, options [mss 1460,nop,wscale 6], length 0 > 00:00:00.000788 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [.], ack 1, win 1026, length 0 > 00:00:00.002003 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [.], ack 1, win 127, length 0 > 00:00:00.028959 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [P.], seq 1:318, ack 1, win 1026, length 317 > 00:00:00.029884 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [.], ack 318, win 108, length 0 > 00:00:00.029925 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [.], seq 1:4097, ack 318, win 108, length 4096 Amazon's SSL certificate is too large to fit into their initial send window, and despite the fact that FreeBSD has received 4096 bytes (more than 2 MSS, and thus enough that it SHOULD send an ACK according to RFC 2581) we delay that ACK here. 100 ms later, our delayed-ACK timer fires, we send the ACK, Amazon's TCP stack finishes sending their SSL certificate, and everything starts moving again. > 00:00:00.129497 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [.], ack 4097, win 1026, length 0 > 00:00:00.130258 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [P.], seq 4097:4241, ack 318, win 108, length 144 > 00:00:00.131332 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [P.], seq 318:632, ack 4241, win 1026, length 314 > 00:00:00.136398 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [P.], seq 4241:4288, ack 632, win 128, length 47 > 00:00:00.136773 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [P.], seq 632:1011, ack 4288, win 1026, length 379 > 00:00:00.141006 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [F.], seq 4867, ack 1011, win 144, length 0 > 00:00:00.141022 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [.], ack 4288, win 1026, length 0 > 00:00:00.141033 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [P.], seq 4288:4867, ack 1011, win 144, length 579 > 00:00:00.141059 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [.], ack 4868, win 1017, length 0 > 00:00:00.141167 IP 10.142.128.54.13252 > 176.32.98.166.443: Flags [P.], seq 1011:1038, ack 4868, win 1026, length 27 > 00:00:00.142036 IP 176.32.98.166.443 > 10.142.128.54.13252: Flags [R], seq 3035214568, win 9700, length 0 Out of 142 ms that this TCP connection is alive, 100 ms was wasted. This seems like something which ought to be fixed... -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid