From owner-freebsd-net@FreeBSD.ORG Wed Mar 12 08:49:14 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08EA21065675 for ; Wed, 12 Mar 2008 08:49:14 +0000 (UTC) (envelope-from rizzojake@gmail.com) Received: from po-out-1718.google.com (po-out-1718.google.com [72.14.252.154]) by mx1.freebsd.org (Postfix) with ESMTP id C140B8FC38 for ; Wed, 12 Mar 2008 08:49:13 +0000 (UTC) (envelope-from rizzojake@gmail.com) Received: by po-out-1718.google.com with SMTP id y22so4645860pof.3 for ; Wed, 12 Mar 2008 01:49:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=jw/lL02m0hwHN/58emHntWGd1KM35BuQ9hZCurtaaK0=; b=su+X8Uvk5lGF/4slyM7iRpPXnucGq0wo/107pyXQZZRLlTW35HMWo5w8dzYb2QmotTwybTXaq5Ok78lnvVZ3koxXpqaQF4Czx8K2TV54ia8ku03nUP7I4nnQU1CLLNZwKxJwFqow2GzpQtxSIbsGMjnDTXuA9u7aWEHLR/zbLFw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=pCl5vvNPDTObD3DLUSoahCv8XmwLfsuUbh4+GcIXPKoKdrcrd9nmMo5D1GKBkABIvvJ0kiXIPjSRVZtxOPhMXygWM8zDKN4JL+cpZ9yuwyi+6AJEwJ6NtAGiOb6JYA+28F/aQZZ0qLV5N6HFpPz0p/NL0xjH/wCR8CWuACgSH1I= Received: by 10.141.171.6 with SMTP id y6mr4960415rvo.84.1205311752739; Wed, 12 Mar 2008 01:49:12 -0700 (PDT) Received: by 10.141.22.11 with HTTP; Wed, 12 Mar 2008 01:49:12 -0700 (PDT) Message-ID: Date: Wed, 12 Mar 2008 08:49:12 +0000 From: "Jake Rizzo" To: "d.s. al coda" In-Reply-To: MIME-Version: 1.0 References: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Kip Macy , freebsd-net@freebsd.org Subject: Re: TCP options order changed in FreeBSD 7, incompatible with some routers X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 08:49:14 -0000 Exact same problem that i'm having. I confirmed it exists in 7.0 only since downgrading one of our servers back to 6.3 stable allowed the same clients to connect again. This seems to work for us as a workaround: sysctl net.inet.tcp.sack.enable=0 On 3/12/08, d.s. al coda wrote: > > On 3/11/08, Kip Macy wrote: > > > Are you running 7.0-RELEASE? What I believe was this issue was a > > showstopper for it, so I'm surprised to hear of it now. > > > > -Kip > > > > > Yes, we are running 7.0-RELEASE. > > -coda > > > On Tue, Mar 11, 2008 at 5:56 PM, d.s. al coda > > wrote: > > > Hi, > > > We recently upgraded one of our webservers to FreeBSD 7, and we > started > > > receiving complaints from some users not able to connect to that > server > > > anymore. On top of that, users were saying that the problem only > > occurred on > > > Windows (at least, the ones who had more than on OS to try it out). > > > > > > After managing to get a user who had the problem running windump, > > running > > > tcpdump on the new server, and comparing that to the windump & > tcpdump > > > output for a "control" user (me) that could connect, we managed to > > figure > > > out the following: > > > - For the user with this problem, ping works fine, but all TCP > > connections > > > to the server fail. > > > - The user, trying to connect, sends out a SYN packet, receives no > > response, > > > and retries a few times until timing out. > > > - The server sees a bunch of SYN packets and responds with SYN-ACK > each > > > time. > > > - The issue only seems to arise if the sender has RFC1323 disabled. > > > > > > So, the SYN-ACK is getting lost somewhere. > > > > > > - For the control user (who can connect via TCP just fine), we set > the > > TCP > > > window size and RFC1323 options the same as the user with the > problem. > > > - The control user sees the SYN-ACK packet. > > > - We send a connection attempt to one of our other servers, running > > FreeBSD > > > 5.5, and one to the server running FreeBSD 7. > > > - There is only one notable difference between the responses: the > order > > of > > > the options. > > > - FreeBSD 5.5 has > > > - FreeBSD 7 has (there is of course an > aligning > > nop > > > after the eol, which tcpdump skips) > > > - These options don't appear in this exact configuration when using > > RFC1323 > > > options. > > > > > > I get a hunch that the users with the problem have a router that > > erroneously > > > thinks that these options are invalid, or thinks that the some part > of > > byte > > > sequence (e.g. 0204 05b4 0101 0402) is an attack. > > > > > > Just to try it out, I patched tcp_output.c so that the SACK permitted > > option > > > was aligned on a 4-byte boundary, preventing the "sackOK, eol" > pattern > > from > > > ever occuring. Looking through previous versions, I found where the > > tcp > > > option code had changed, and there used to be a comment about putting > > SACK > > > permitted last, but I can't tell if it's relevant. > > > > > > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_output.c.diff?r1=1.125;r2=1.126 > > > > > > The one-line patch to tcp_output.c is attached. > > > > > > Sure enough, it fixed the problem. Afterwards, we collected some > > information > > > about the routers the users who had the problem were using, and while > > they > > > didn't all have the same manufacturer, several mentioned that their > > router > > > had a built-in firewall, which, when they disabled it, allowed them > to > > > access the server. > > > > > > Does all of this sound reasonable? And if so, would it be worth > > submitting > > > this patch? I don't know if this particular change in options order > was > > > intentional, or just a side-effect of the new code, but it certainly > > works > > > around an extremely hard-to-diagnose problem. > > > > > > -coda > > > > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >