From owner-freebsd-net@freebsd.org Mon Jul 8 10:38:02 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E675615DAC32 for ; Mon, 8 Jul 2019 10:38:01 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 40DB096DEE for ; Mon, 8 Jul 2019 10:38:00 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id:Cc:To :Subject:From:Date:Sender:Reply-To:Content-ID:Content-Description:Resent-Date :Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post: List-Owner:List-Archive; bh=h18U2K79tsFA52TfwAIX83Z2HrSK+7wn9MaSl+27u6U=; b=q xbG3Pa12V44ohQTt8wLXaO3cWoCzZgkOyEHkC3bFOFXoiYpg7EKDFBBEN+m6o94LoQw8APOKjyYJB X1+BzQjyb+IfBt49kKLdjJblGPa+PjYO/S6ntCgNUFfE17Go3DUN9OpfIpeCIdT65u/lmoooaghVv 7pxdJ+use2FMJNwo=; Received: from [10.10.10.39] (helo=frv39.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1hkR1q-000Ane-Uj for freebsd-net@freebsd.org; Mon, 08 Jul 2019 13:37:50 +0300 Date: Mon, 08 Jul 2019 13:37:50 +0300 From: Paul Subject: Issues with TCP Timestamps allocation To: Michael Tuexen , freebsd-net@freebsd.org Received: from devgs@ukr.net by frv39.fwdcdn.com; Mon, 08 Jul 2019 13:37:50 +0300 Message-Id: <1562579483.67527000.24rw4xi5@frv39.fwdcdn.com> X-Mailer: mail.ukr.net 5.0 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-Rspamd-Queue-Id: 40DB096DEE X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ukr.net header.s=ffe header.b=q xbG3Pa; dmarc=pass (policy=none) header.from=ukr.net; spf=pass (mx1.freebsd.org: domain of devgs@ukr.net designates 212.42.77.196 as permitted sender) smtp.mailfrom=devgs@ukr.net X-Spamd-Result: default: False [-5.53 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ukr.net:s=ffe]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:212.42.77.0/24]; FREEMAIL_FROM(0.00)[ukr.net]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.63)[ipnet: 212.42.77.0/24(-4.55), asn: 8856(-3.69), country: UA(0.08)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_SPAM_SHORT(0.11)[0.112,0]; DWL_DNSWL_LOW(-1.00)[ukr.net.dwl.dnswl.org : 127.0.5.1]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ukr.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[ukr.net,none]; MX_GOOD(-0.01)[mxs.ukr.net]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[ukr.net]; ASN(0.00)[asn:8856, ipnet:212.42.77.0/24, country:UA]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jul 2019 10:38:02 -0000 Hi team, Recently we had an upgrade to 12 Stable. Immediately after, we have started seeing some strange connection establishment timeouts to some fixed number of external (world) hosts. The issue was persistent and easy to reproduce. Thanks to a patience and dedication of our system engineer we have tracked this issue down to a specific commit: https://svnweb.freebsd.org/base?view=revision&revision=338053 This patch was also back-ported into 11 Stable: https://svnweb.freebsd.org/base?view=revision&revision=348435 Among other things this patch changes the timestamp allocation strategy, by introducing a deterministic randomness via a hash function that takes into account a random key as well as source address, source port, dest address and dest port. As the result, timestamp offsets of different tuples (SA,SP,DA,DP) will be wildly different and will jump from small to large numbers and back, as long as something in the tuple changes. After performing various tests of hosts that produce the above mentioned issue we came to conclusion that there are some interesting implementations that drop SYN packets with timestamps smaller than the largest timestamp value from streams of all recent or current connections from a specific address. This looks as some kind of SYN flood protection. To ensure that each external host is not going to see a wild jumps of timestamp values I propose a patch that removes ports from the equation all together, when calculating the timestamp offset: Index: sys/netinet/tcp_subr.c =================================================================== --- sys/netinet/tcp_subr.c (revision 348435) +++ sys/netinet/tcp_subr.c (working copy) @@ -2224,7 +2224,22 @@ uint32_t tcp_new_ts_offset(struct in_conninfo *inc) { - return (tcp_keyed_hash(inc, V_ts_offset_secret)); + /* + * Some implementations show a strange behaviour when a wildly random + * timestamps allocated for different streams. It seems that only the + * SYN packets are affected. Observed implementations drop SYN packets + * with timestamps smaller than the largest timestamp value of all + * recent or current connections from specific a address. To mitigate + * this we are going to ensure that each host will always observe + * timestamps as increasing no matter the stream: by dropping ports + * from the equation. + */ + struct in_conninfo inc_copy = *inc; + + inc_copy.inc_fport = 0; + inc_copy.inc_lport = 0; + + return (tcp_keyed_hash(&inc_copy, V_ts_offset_secret)); } /* In any case, the solution of the uptime leak, implemented in rev338053 is not going to suffer, because a supposed attacker is currently able to use any fixed values of SP and DP, albeit not 0, anyway, to remove them out of the equation. There is the list of example hosts that we were able to reproduce the issue with: curl -v http://88.99.60.171:80 curl -v http://163.172.71.252:80 curl -v http://5.9.242.150:80 curl -v https://185.134.205.105:443 curl -v https://136.243.1.231:443 curl -v https://144.76.196.4:443 curl -v http://94.127.191.194:80 To reproduce, call curl repeatedly with a same URL some number of times. You are going to see some of the requests stuck in `* Trying XXX.XXX.XXX.XXX...` For some reason, the easiest way to reproduce the issue is with nc: $ echo "foooooo" | nc -v 88.99.60.171 80 Only a few such calls are required until one of them is stuck on connect(): issuing SYN packets with an exponential backoff.