From owner-freebsd-net@freebsd.org Mon Jul 8 15:22:43 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE1D715E14E5 for ; Mon, 8 Jul 2019 15:22:42 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D14E573629 for ; Mon, 8 Jul 2019 15:22:41 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id: References:In-Reply-To:Cc:To:Subject:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=9dvi4ZtW9kSLFJrtLyS7tEGPO/k5vmcMAbwvDMBhl/M=; b=m0GOmK4uoHVPIODMMGQ71waKuM v6d30CFeJ0nbdB5g3M4e1syVgeYD4aaRyizUSqei6SKyvVHwvcHWZGYYkJ2SLPTVj1L8BuULG3jGa cA28Aj9mrQT57xdMXXdSUyW8gX3LBM7klpFkU3ddpjel84G8b3hqqa7NTXS5NJw3Chok=; Received: from [10.10.10.39] (helo=frv39.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1hkVTS-000PyN-Qa for freebsd-net@freebsd.org; Mon, 08 Jul 2019 18:22:38 +0300 Date: Mon, 08 Jul 2019 18:22:38 +0300 From: Paul Subject: Re[2]: Issues with TCP Timestamps allocation To: Michael Tuexen Cc: freebsd-net@freebsd.org Received: from devgs@ukr.net by frv39.fwdcdn.com; Mon, 08 Jul 2019 18:22:38 +0300 In-Reply-To: References: <1562579483.67527000.24rw4xi5@frv39.fwdcdn.com> <32FD061B-245C-41D2-81DE-1B4756A7173D@freebsd.org> <1562591379.369129000.gpmxvurq@frv39.fwdcdn.com> X-Reply-Action: reply Message-Id: <1562599181.734953000.1l9a1d23@frv39.fwdcdn.com> X-Mailer: mail.ukr.net 5.0 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-Rspamd-Queue-Id: D14E573629 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ukr.net header.s=ffe header.b=m0GOmK4u; dmarc=pass (policy=none) header.from=ukr.net; spf=pass (mx1.freebsd.org: domain of devgs@ukr.net designates 212.42.77.196 as permitted sender) smtp.mailfrom=devgs@ukr.net X-Spamd-Result: default: False [-6.51 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ukr.net:s=ffe]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:212.42.77.0/24]; FREEMAIL_FROM(0.00)[ukr.net]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.65)[ipnet: 212.42.77.0/24(-4.61), asn: 8856(-3.73), country: UA(0.08)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DWL_DNSWL_LOW(-1.00)[ukr.net.dwl.dnswl.org : 127.0.5.1]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ukr.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[ukr.net,none]; MX_GOOD(-0.01)[cached: mxs.ukr.net]; NEURAL_HAM_SHORT(-0.85)[-0.852,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[ukr.net]; ASN(0.00)[asn:8856, ipnet:212.42.77.0/24, country:UA]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jul 2019 15:22:43 -0000 8 July 2019, 17:12:21, by "Michael Tuexen" : > > On 8. Jul 2019, at 15:24, Paul wrote: > > > > Hi Michael, > > > > 8 July 2019, 15:53:15, by "Michael Tuexen" : > > > >>> On 8. Jul 2019, at 12:37, Paul wrote: > >>> > >>> Hi team, > >>> > >>> Recently we had an upgrade to 12 Stable. Immediately after, we have started > >>> seeing some strange connection establishment timeouts to some fixed number > >>> of external (world) hosts. The issue was persistent and easy to reproduce. > >>> Thanks to a patience and dedication of our system engineer we have tracked > >>> this issue down to a specific commit: > >>> > >>> https://svnweb.freebsd.org/base?view=revision&revision=338053 > >>> > >>> This patch was also back-ported into 11 Stable: > >>> > >>> https://svnweb.freebsd.org/base?view=revision&revision=348435 > >>> > >>> Among other things this patch changes the timestamp allocation strategy, > >>> by introducing a deterministic randomness via a hash function that takes > >>> into account a random key as well as source address, source port, dest > >>> address and dest port. As the result, timestamp offsets of different > >>> tuples (SA,SP,DA,DP) will be wildly different and will jump from small > >>> to large numbers and back, as long as something in the tuple changes. > >> Hi Paul, > >> > >> this is correct. > >> > >> Please note that the same happens with the old method, if two hosts with > >> different uptimes are bind a consumer grade NAT. > > > > If NAT does not replace timestamps then yes, it should be the case. > > > >>> > >>> After performing various tests of hosts that produce the above mentioned > >>> issue we came to conclusion that there are some interesting implementations > >>> that drop SYN packets with timestamps smaller than the largest timestamp > >>> value from streams of all recent or current connections from a specific > >>> address. This looks as some kind of SYN flood protection. > >> This also breaks multiple hosts with different uptimes behind a consumer > >> level NAT talking to such a server. > >>> > >>> To ensure that each external host is not going to see a wild jumps of > >>> timestamp values I propose a patch that removes ports from the equation > >>> all together, when calculating the timestamp offset: > >>> > >>> Index: sys/netinet/tcp_subr.c > >>> =================================================================== > >>> --- sys/netinet/tcp_subr.c (revision 348435) > >>> +++ sys/netinet/tcp_subr.c (working copy) > >>> @@ -2224,7 +2224,22 @@ > >>> uint32_t > >>> tcp_new_ts_offset(struct in_conninfo *inc) > >>> { > >>> - return (tcp_keyed_hash(inc, V_ts_offset_secret)); > >>> + /* > >>> + * Some implementations show a strange behaviour when a wildly random > >>> + * timestamps allocated for different streams. It seems that only the > >>> + * SYN packets are affected. Observed implementations drop SYN packets > >>> + * with timestamps smaller than the largest timestamp value of all > >>> + * recent or current connections from specific a address. To mitigate > >>> + * this we are going to ensure that each host will always observe > >>> + * timestamps as increasing no matter the stream: by dropping ports > >>> + * from the equation. > >>> + */ > >>> + struct in_conninfo inc_copy = *inc; > >>> + > >>> + inc_copy.inc_fport = 0; > >>> + inc_copy.inc_lport = 0; > >>> + > >>> + return (tcp_keyed_hash(&inc_copy, V_ts_offset_secret)); > >>> } > >>> > >>> /* > >>> > >>> In any case, the solution of the uptime leak, implemented in rev338053 is > >>> not going to suffer, because a supposed attacker is currently able to use > >>> any fixed values of SP and DP, albeit not 0, anyway, to remove them out > >>> of the equation. > >> Can you describe how a peer can compute the uptime from two observed timestamps? > >> I don't see how you can do that... > > > > Supposed attacker could run a script that continuously monitors timestamps, > > for example via a periodic TCP connection from a fixed local port (eg 12345) > > and a fixed local address to the fixed victim's address and port (eg 80). > > Whenever large discrepancy is observed, attacker can assume that reboot has > > happened (due to V_ts_offset_secret re-generation), hence the received > > timestamp is considered an approximate point of reboot from which the uptime > > can be calculated, until the next reboot and so on. > Ahh, I see. The patch we are talking about is not intended to protect against > continuous monitoring, which is something you can always do. You could even > watch for service availability and detect reboots. A change of the local key > would also look similar to a reboot without a temporary loss of connectivity. > > Thanks for the clarification. > > > >>> > >>> There is the list of example hosts that we were able to reproduce the > >>> issue with: > >>> > >>> curl -v http://88.99.60.171:80 > >>> curl -v http://163.172.71.252:80 > >>> curl -v http://5.9.242.150:80 > >>> curl -v https://185.134.205.105:443 > >>> curl -v https://136.243.1.231:443 > >>> curl -v https://144.76.196.4:443 > >>> curl -v http://94.127.191.194:80 > >>> > >>> To reproduce, call curl repeatedly with a same URL some number of times. > >>> You are going to see some of the requests stuck in > >>> `* Trying XXX.XXX.XXX.XXX...` > >>> > >>> For some reason, the easiest way to reproduce the issue is with nc: > >>> > >>> $ echo "foooooo" | nc -v 88.99.60.171 80 > >>> > >>> Only a few such calls are required until one of them is stuck on connect(): > >>> issuing SYN packets with an exponential backoff. > >> Thanks for providing an end-point to test with. I'll take a look. > >> Just to be clear: You are running a FreeBSD client against one of the above > >> servers and experience the problem with the new timestamp computations. > >> > >> You are not running arbitrary clients against a FreeBSD server... > > > > We are talking about FreeBSD being the client. Peers that yield this unwanted > > behaviour are unknown. Little bit of tinkering showed that some of them run > > Debian: > > > > telnet 88.99.60.171 22 > > Trying 88.99.60.171... > > Connected to 88.99.60.171. > > Escape character is '^]'. > > SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3 > Also some are hosted by Hetzner, but not all. I'll will look into > this tomorrow, since I'm on a deadline today (well it is 2am tomorrow > morning, to be precise)... Thanks a lot, I would appreciate that. > > Best regards > Michael > > > > > >> > >> Best regards > >> Michael > >> > >> > >