From owner-freebsd-net@FreeBSD.ORG Fri Feb 1 22:54:47 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 71BC637D for ; Fri, 1 Feb 2013 22:54:47 +0000 (UTC) (envelope-from kevin@your.org) Received: from mail.your.org (mail.your.org [IPv6:2001:4978:1:2::cc09:3717]) by mx1.freebsd.org (Postfix) with ESMTP id 3B2D32A8 for ; Fri, 1 Feb 2013 22:54:47 +0000 (UTC) Received: from mail.your.org (chi02.mail.your.org [204.9.55.23]) by mail.your.org (Postfix) with ESMTP id C12EBF06C5F; Fri, 1 Feb 2013 22:54:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=your.org; h=content-type :mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s= selector1; bh=A4fNsr5EwB4QDtRQ0KTHiGDd9dA=; b=bOj7wFOlqvkKDdnpJw iFv/gGvXkbk0KyEQ4oKtVaLgpVy1zm3C50/oPLzqL+PAz/LWEAjUpM6yu7+akoBl AwyEnd6CI19rInVa006hChBKefj6gbrxBLsYr6zxyzDRloRRRIRpOIQ/R+bx6dxs EMpo63hHaSlb6cWmx4Vtgtkpc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=your.org; h=content-type :mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; q=dns; s= selector1; b=aX4BhL7D6ZEjBLMdWNR1r5vqMNLD9WyGGlN30eVI4PhoofFeLhd auA+KY40XcE4GQIzRVYaOi1shEyTefiqx+0EaE9XGpo3c1PmK+DhdVZAVg4aijwD gBC8V3JCuKRInOZqOosRNTHSsWRUqkcGUBfhJfPEBx+8UZU4+C6O5AnQ= Received: from vpn132.rw1.your.org (vpn132.rw1.your.org [204.9.51.132]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.your.org (Postfix) with ESMTPSA id 13D2CF06C63; Fri, 1 Feb 2013 22:54:45 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Syncookies break with Windows 8 From: Kevin Day In-Reply-To: <510C4424.4030701@networx.ch> Date: Fri, 1 Feb 2013 16:54:43 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <510C4424.4030701@networx.ch> To: Andre Oppermann X-Mailer: Apple Mail (2.1499) Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2013 22:54:47 -0000 On Feb 1, 2013, at 4:39 PM, Andre Oppermann = wrote: >=20 > This is not true. FreeBSD uses bits in the timestamp to encode all > recognized TCP options including window scaling. >=20 Sorry, you are correct here. Reading through a half dozen TCP = implementations in the last day trying to figure out what was happening = and it all turns into a blur. :) >> This doesn't affect Linux because it uses timestamp options to stuff = the client's wscale, so it >> gets re-learned on the ACK. OpenBSD and OS X don't have syncookies. = NetBSD seems to have the same >> problem if it's new syncookie implementation gets turned on. >=20 > This can't be because of the lack of timestamps. Linux must be > encoding the scale in the ISS taking away bits from the cookie. >=20 > I haven't looked into how Linux actually does it recently. >=20 In Linux, it appears if timestamps are not enabled, either because = they're turned off by sysctl or the server didn't see one from the = client, it disables scaling. http://fxr.watson.org/fxr/source/net/ipv4/syncookies.c?v=3Dlinux-2.6#L235 wscale_ok defaults to 0. It's set to '1' only if the server saw_tstamp, = sysctl_tcp_timestamps isn't turned off, etc.=20 >> Any thoughts? Was there a reason why we're forcing the use of wscale = on syncookie connections? >=20 > We can change the behavior of syncookies in a couple of ways to > deal with this problem: >=20 > 1/ send syncookies only when the syncache overflows and set wscale > to 0 in the SYN-ACK when timestamps are not active. >=20 That sounds similar to Linux, if I'm reading the code correctly. > At the moment we send syncookies on every SYN-ACK and bump the oldest > entry from the syncache when it is full. That results in potentially > every segment degrading to syncookie only. The default values are > insufficient for such high loads. >=20 > In general at 10,000 connections per second you should significantly > increase the size of your syncache to 3 * conn/sec at least. >=20 > In the loader you can set these tunables: >=20 > net.inet.tcp.syncache.hashsize =3D 2048 > net.inet.tcp.syncache.bucketlimit =3D 32 > net.inet.tcp.syncache.cachelimit =3D 65536 >=20 > These settings are a bit more complicated than they should be. >=20 We're doing similar. During a DDoS we tried changing to syncookies_only = and somehow that got left that way, which is how we ran into this = problem more reliably. Turning that off, and using higher syncache sizes = helped hide this greatly, but it's probably still happening here and = elsewhere occasionally. -- Kevin