From owner-freebsd-transport@freebsd.org Mon May 9 16:16:21 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9F3BB335B0 for ; Mon, 9 May 2016 16:16:21 +0000 (UTC) (envelope-from jtl@freebsd.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 7170D164F for ; Mon, 9 May 2016 16:16:21 +0000 (UTC) (envelope-from jtl@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6D103B335AF; Mon, 9 May 2016 16:16:21 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CA68B335AE for ; Mon, 9 May 2016 16:16:21 +0000 (UTC) (envelope-from jtl@freebsd.org) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0111.outbound.protection.outlook.com [207.46.100.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 084A4164E for ; Mon, 9 May 2016 16:16:20 +0000 (UTC) (envelope-from jtl@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=junipernetworks.onmicrosoft.com; s=selector1-junipernetworks-onmicrosoft-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=INZJy60l9M4qphSC2WJuwc/Ozp2mqWklw14saGpA0MM=; b=SurxUPMtyrgoEUeB3dwJcj3oC/neVky/LGHRSl7s8Sh8maTZ3LalCIE2DX9nFh/fdx/mLBVVwQiUcTwUGa3SxcGKsPl/0ZHXBspyh5RfpZK80fhBZiugCdpjGAEv1QlM2+uP8n825g6X6P9VjDYIzau9uaGm3cgz3lEM7m+ohEA= Received: from BY1PR0501CA0012.namprd05.prod.outlook.com (10.162.139.22) by CY1PR05MB2492.namprd05.prod.outlook.com (10.167.10.25) with Microsoft SMTP Server (TLS) id 15.1.492.11; Mon, 9 May 2016 15:43:22 +0000 Received: from BN1BFFO11FD037.protection.gbl (2a01:111:f400:7c10::1:173) by BY1PR0501CA0012.outlook.office365.com (2a01:111:e400:4821::22) with Microsoft SMTP Server (TLS) id 15.1.492.11 via Frontend Transport; Mon, 9 May 2016 15:43:22 +0000 Authentication-Results: spf=softfail (sender IP is 66.129.239.18) smtp.mailfrom=freebsd.org; neville-neil.com; dkim=none (message not signed) header.d=none;neville-neil.com; dmarc=none action=none header.from=freebsd.org; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning freebsd.org discourages use of 66.129.239.18 as permitted sender) Received: from p-emfe01a-sac.jnpr.net (66.129.239.18) by BN1BFFO11FD037.mail.protection.outlook.com (10.58.144.100) with Microsoft SMTP Server (TLS) id 15.1.492.8 via Frontend Transport; Mon, 9 May 2016 15:43:21 +0000 Received: from magenta.juniper.net (172.17.27.123) by p-emfe01a-sac.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.123.3; Mon, 9 May 2016 08:17:55 -0700 Received: from [172.29.37.47] ([172.29.37.47]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id u49FHrZ36195; Mon, 9 May 2016 08:17:53 -0700 (PDT) (envelope-from jtl@freebsd.org) User-Agent: Microsoft-MacOutlook/f.15.1.160411 Date: Mon, 9 May 2016 11:17:52 -0400 Subject: Re: Patches to improve SYN performance when under attack From: "Jonathan T. Looney" Sender: Jonathan Looney To: George Neville-Neil , "transport@freebsd.org" Message-ID: <39D14339-F093-4A0C-9A35-27A6D8D43A5F@juniper.net> Thread-Topic: Patches to improve SYN performance when under attack References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.18; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(11905935001)(189002)(199003)(24454002)(377454003)(9170700001)(9686002)(586003)(561924002)(81166005)(5001770100001)(4001350100001)(107886002)(54356999)(50986999)(76176999)(83506001)(8936002)(47776003)(2906002)(19580405001)(230700001)(33656002)(87936001)(50466002)(1220700001)(19580395003)(105596002)(106466001)(23676002)(2950100001)(77096005)(15975445007)(11100500001)(36756003)(86362001)(16796002)(189998001)(2501003)(6806005)(92566002)(5820100001)(5890100001)(42262002)(104396002)(349545003); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR05MB2492; H:p-emfe01a-sac.jnpr.net; FPR:; SPF:SoftFail; MLV:sfv; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11FD037; 1:IExN45ExMkxaHud2mUizhihhEdv5wySLidZ7L1/QI2okklgUR1pPotFSDnc1qWsc8iSPjNrA4Ov7jrubrzDDDxGoJxL6FU5IXWLDC9cCSw4HdGEJoU6tIMOemlp9s4D6/CYHyiQGoHl/MLbKjds1H+IP9usr/ht7YzCsfu54oFYBxCJtfyApN2sTh84phRySlv67Ai5wFvbqDCm2Rf8UCCUjZm52tAA+Jon7LGpMibOdv7F6LVi7BMSmVpB5iOdTwWKzW0LWfbBQ9PG8mRGMEOQ2AmSWckWILZ8zssEgA6Rwnp4F843Ar9Bd1MmKuxVV6yBUO4kfK6NPZC2q807JjVRZ/T/1QXBAh2mDjKdo5WhaOYpoMRE7ERM4Szb5zvZCTIfRxc7XXJcrkxUOsaI7HJ073hjHXDWJs4HvPx2FZPVsHAAFyWauMPmy9991Mw32Wo5xuSKA4Hb90/3mlIFwkh1EPNAA7qDlbEy+Jjy7ogoTBVvgFZJz8Px7EAjr1Heo X-MS-Office365-Filtering-Correlation-Id: f33bc3ab-7403-4fd7-cc66-08d37820a654 X-Microsoft-Exchange-Diagnostics: 1; CY1PR05MB2492; 2:DOB+WKtQ8vTEjsnNv5t7gq/jUROWmC3uZRcx0oo7hYdiRhWyHB7w7uxMiKBpMH+pMIIWENJFT8LRK1b+J8Cx8tNXAU+2zFGK7kAdn4e98x/gOqogthqa1BIZId6iQZPTqIa4UjQd64yVW9tl38Gcz5CA2CeHNJB0S4imRmhuhga3fKGNsZTevbbjiBWJHN0z; 3:MjTsggsDWUY+SdM3p/j9O3zK4F2YxB9CU6wDrV5VcAi03mWgHaRFWFXMuRYusOj+LZaycb9ZCBrt13BqsarCUcSsNY7ktXPIuYD7EdDIWqv0/rWZuI7wboJ5bSSmCcs+v/hhnjYYhIhbAZ7QWEi16f8avDRn1dRzxm9BF3k8zXPPLnvOB9+SdtYhuNO0wCQZiXsGtGD/o2I+8IIhQub7id1MFshmc2djRjpw4EUhgd0=; 25:Q0fjlw82lN63u8QcdajUz0IpkdOn3+hF9M2L/fIMl3PVnDRjLroRxqOQBFSYR6v8FxtL50tkePUAtCnEouhtlCxILmSB2XAMqWwJrl7C4fMNuWbeovV/iVu0hMwGzLg4KTotMIGR17YTe9ND5s0UmzKQ+iD8qkB+4wxorEhCWZNwN4VAhuUkdnGgIxACQXcxhTcwvUknozTlBsICmK76AO/6KCApGhYbd09qFZzrQ1MR+vZlFl+hqSqfdsb7FhJaJzflB15fFHK8IoSK+bZZgZ9Tj3zkiyarSk8BSJqTS3Vbk85zCz/NRhPyi45sZHlWnSsdtH+QcJjib7CJRrBRffVMjsd6z50ui98hRJJzauhz0qbcHcvIZDnGOJymf1b3PjPp79t/M7OwNu95mt7XDk8POzOM+8A8kD/gX4JMZhQJkko0bUpq8fCXxgsnOMqG X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR05MB2492; X-JNPR-Received-From: outside X-Microsoft-Exchange-Diagnostics: 1; CY1PR05MB2492; 20:9U3UG/Dzui9sMf+vUyLJcJEPc4/eFMMWuejFuWjK2LX3XN0YMekL8gS8FtimTWvC5ZYb3euWHpzWpl1oyLOY4lvIxwucZHrEpUY2oyiJi/TrZvrUiGpCnq5J7Ap8gXy5uRS8+EJpcHFWcAcAtgns2qVlFTMoAuiZK4uBGZY1hPYblnC52HcecfO91iovx3H5qVIxqgVGLMAwhE+e4ukHKqYjtEuSWpikFfnhpksYnWQBca79BkS1SGWjs/UbRhFYGlB5s+dmjTN/RwUF7/wu+P9T9quDjEgQzQv0ComtnqcTfwNedIF9vFC1KfmIuiEO3M6gDhjNWB1iuFQdCOsrupdtZsYEflHMpJSOkNDqDHmUUkTMXvJpqfY05PE7TaSii18ZFbMbrTXpwhIxKKHDiZF1YSzdrsn5Q+6UQTTvTozCf26vfj5gqHq60S9f5Q8BNs6upA+Roczl8oiSJugyrfqvf7oa/nmwT2DjrvEeqyZTG78/6mD51R+wq4sm5gUs X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(13024025)(13023025)(13015025)(13018025)(13017025)(5005006)(8121501046)(3002001)(10201501046)(6055026); SRVR:CY1PR05MB2492; BCL:0; PCL:0; RULEID:; SRVR:CY1PR05MB2492; X-Microsoft-Exchange-Diagnostics: 1; CY1PR05MB2492; 4:6G7qw6VyKvcXdjjCBPrkm6CI9fprtZ+x+ANcV+c7TmIr1tZDPboZHp6vPQ+fUbh3e7CI+cD2yQdytWi869AUeMemZ3+vnRBM388KrU6zSQftfZe1oZWyV7sWOqnhk4I28e0OBjYSCOevflqRNPsv/pLAterr6bF9ojYcn1BsSNJ5Hd1VcvrGnOKdND5Cph+F0u2Qjn4yYhgzHHfy2OQCBDxuhb/e5RmA7ucJepIO5ODq/OvJF2IMk0cOItzI8DWpUE7fuSv5iVUSWjPFxKEsweGb0Rr9oS8hdZLC2qi5ZbLrO/yRGM6oaN6tQxt6WaCnA4DJW9RI/ienVoXgfAovZ6kIIGdtzZWwNvBMrCdGkXW/41dJ1fsReHMDS7ItAvCQXA0E2/X1ZIoPhU2Uaq8mdyTjLw8I6S7hovfE+TaDMet+kWN7RuH9rkV/W8OBrwukM7ggqIObP0VebZiRAh6XNqGQTF0CaQNpVNDI1hs4ois= X-Forefront-PRVS: 0937FB07C5 X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTFQUjA1TUIyNDkyOzIzOlk4bFJhNE1Qa3hEaVE0aHJrdDU3UWE2b0Zm?= =?utf-8?B?bnRwdDNJbEdRZ2lZc2tlejlXbW5talR0S3NZMXB4L0ZQdG1aUG9rM2h4QXZh?= =?utf-8?B?Y0tRM2c0MjU4QzlvdHhJaWlYOENKcVN4bXh2RjlNL3l2RzJyU09qaENKR3VQ?= =?utf-8?B?RkY4VjhGWGJYTTJaWnc0eGFZMmF5OWhDa3JUbjVIWFBSM1JvS09iM2ROalNq?= =?utf-8?B?MFJqeHZCQ1FWWXUyUHU5UCtITGtBZlNGZjJRL0NtWGNsa3hEd0wzS3E0eEZ6?= =?utf-8?B?T3JzNGVsSllJRUhaVGtaT3E2enJHVWw0MEU3TkQrTVBIb1QyUU1hTjArK2ZY?= =?utf-8?B?eWVLeWJYZXNCUUJFd05zZTZKdkRXdng5TmJsOU95ZlhKd2F0ZU1tZTBES2Zo?= =?utf-8?B?RHdKcTFHMDZYV2JsaytiWCtnM0NYSXBPb1VRb2tiSTJma1dxMUJQd2VUcVMw?= =?utf-8?B?N0swWUt2Q0cvTTRsbEdSL3V0QWNrVCthbjFVdnVSTUh2NkxTT3JXTzRqL0Nr?= =?utf-8?B?Rk9wWmh2R2NiM0dYalVtR0FXOThpMDFXMFFwbmNIUFd1dWViTDlCZDEzUG1Y?= =?utf-8?B?ZUpPSzhBM1dielJYSEgweVNKMHdtcnNpc1hkY0VFOFB6QTZHREpWSXJhM0tq?= =?utf-8?B?WDRrRm1TeWMxL3h2ZW05U25yUUoza0dmeldqbVIwdVNZOGN0QUI4SGJqcG55?= =?utf-8?B?M1k1TVVGdUl3VmNYbGw1NmhiTXFQVUI4Y0VlMWZ6YURhR3VLWkFXZXNlekpF?= =?utf-8?B?dnA0SDZFZDdJb3lnYVVVbERCMzNyMjBzZzg4M0ZNMWhCWitwZE9tRHBidkpj?= =?utf-8?B?NDdjN3g3cjFBZmJYQ3pEeXFYbHd4QTRwTHhDdG8rUEJJRUp1RXRNdkVNTTVK?= =?utf-8?B?cGZqM2hJRk1hUWEwTkV0ZkRMZXJaUmdaUC9tNC9BR2RJWFNNNTBxK2t1a1dW?= =?utf-8?B?bUxBam9MOGpsRVdWUEFnTDZNYXorUzRja0NBNXdWSjcxUWZ6Tzk2V0p5SzVD?= =?utf-8?B?VmNQNVE5bThZb1lwYXJ5MWcxRnFyQ1V2djF2dlNQakVyWjFUcmtEQzZ6cWFk?= =?utf-8?B?ZzE2bTJHN2diK01hVC9hNmtnZTIxNVFBaHU2UmhjYzBOdzZkVjAwcjB1MGRi?= =?utf-8?B?TGtIdUljRjB1VDcwYjJBOWFFVUpsbzZobDY5eU5OMU9aMEZ1dW5ZMUp2SW9G?= =?utf-8?B?N09sSVNIVmVqU3pyVWh3QnVBeGp5MmI0VGY5dUpYWVVNbTZJbUptV3l1SHhP?= =?utf-8?B?cmo1OW1DOVY1YzFudnpoN2xxTVBJMVVRYlJBa3BrNXQzek9PT0tudzljSlM1?= =?utf-8?B?RnFxR3NNTzRtRXcwellOWXFMOU0xQ2w5Tkp3dzZpSmxUNVJXSlhvQmJtWEJU?= =?utf-8?B?eG1vTzFBd1J1ckdXZ3hMTC9uZDVLU0hTQUZuVkd1eGpuUW04NlluWFE0alJX?= =?utf-8?B?RDZkM1dveXpzcHNTNEo4cnE1TnI2b0FpWmFCSWlPTzVMWTBOcUpSM2FyRUYz?= =?utf-8?B?MHJST0l6L1RBQ2l2NnJySk51TmxtSUQyZFJDOGJCQ3pyUy9kMnBLbWRRamUz?= =?utf-8?B?bGRvbmh5eUNYUFo1TUhYelBjc0c5TVFjMDg2S3BpN1U0N1MrUm1mMFZSbDJS?= =?utf-8?B?bUJQYWdmNFF0MEd1bVVUZXNoQjNyU3NFbzZOL21jenJ2N2ZZZTUvYlN6cFZH?= =?utf-8?Q?uBgGq/gRLchBgz4vfs=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR05MB2492; 5:ukOF/gvNdCvWgDnati7oTz5lya+Xp+ihuyl4dOSD+qBzVQl8fzygYQgQH3gxuO4f8YP68U19oCdJ93Jy5PzW9TM2mNu4FR4v1FrXn4tdf5zsNrYsE13IWBfibq7f5mff0m6YB4xzJLFBy2YF5dey/A==; 24:8KkV0+DUQ6uYXKBGRuTx9kV2X/VzsFOecpXfGpw294//cm5Xkcgy1WV09ed1m5EgippZkGOhkbD7PoS12wNEkyYlT2EsCRJB0Q+H3KS2tOs=; 7:tobb6ZkiqNxQLUSXO1zSiKBhDz4LZI6ACG73tT5vFitwHr8zMfE2dZ3g5uxb4I/JmcBI8nKNeOBS7qKf3VEGDY4R5o07TnO7lQhaUNsCBkK/7BI8oVnRacLqi8sdNcVVLp4oxE62q4aT6H5ZORa30xp6zl2tcWtulL6UC8VQ0NC9kcXtcVKmyP7GF+oaXfbq SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2016 15:43:21.0490 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.18]; Helo=[p-emfe01a-sac.jnpr.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR05MB2492 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2016 16:16:21 -0000 On 5/7/16, 11:17 AM, "owner-freebsd-transport@freebsd.org on behalf of George Neville-Neil" wrote: >Can folks take a quick look at these? My quick 2c (since all you asked for was "quick" :-) ): With SYN cookies, I don't think either of these matter. With the syncache, both might. In 11.1, I believe cookies are the default. Jonathan > >Best, >George > > >Forwarded message: > >> From: Robert N. M. Watson >> To: George V. Neville-Neil >> Subject: Fwd: Patches to improve SYN performance when under attack >> Date: Wed, 27 Apr 2016 15:31:34 +0100 >> >> Possibly something for the TCP group to talk about sometime. >> >> Robert >> >>> Begin forwarded message: >>> >>> From: Richard Clayton >>> Subject: Patches to improve SYN performance when under attack >>> Date: 27 April 2016 at 15:20:20 BST >>> To: Robert Watson >>> >>> >>> As discussed, first patch is Oct 2015, second Apr 2016 >>> >>> >>> https://lwn.net/Articles/659199/ >>> >>> This patch series takes the steps to use normal TCP/DCCP ehash >>> table to store SYN_RECV requests, instead of the private per- >>> listener hash table we had until now. >>> >>> SYNACK skb are now attached to their syn_recv request socket, so >>> that we no longer heavily modify listener sk_wmem_alloc. >>> >>> listener lock is no longer held in fast path, including SYNCOOKIE >>> mode. >>> >>> During my tests, my server was able to process 3,500,000 SYN >>> packets per second on one listener and still had available cpu >>> cycles. >>> >>> That is about 2 to 3 order of magnitude what we had with older >>> kernels. >>> >>> https://patchwork.ozlabs.org/patch/610370/ >>> >>> Last known hot point during SYNFLOOD attack is the clearing of >>> rx_opt.saw_tstamp in tcp_rcv_state_process() >>> >>> It is not needed for a listener, so we move it where it matters. >>> >>> Performance while a SYNFLOOD hits a single listener socket went >>> from 5 Mpps to 6 Mpps on my test server (24 cores, 8 NIC RX queues) >>> >>> >>> >>> -- >>> richard @ highwayman . com "Nothing seems the same >>> Still you never see the change from day to day >>> And no-one notices the customs slip away" >> >_______________________________________________ >freebsd-transport@freebsd.org mailing list >https://lists.freebsd.org/mailman/listinfo/freebsd-transport >To unsubscribe, send any mail to "freebsd-transport-unsubscribe@freebsd.org" > From owner-freebsd-transport@freebsd.org Thu May 12 09:50:26 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D270EB37FF4 for ; Thu, 12 May 2016 09:50:26 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id B42261A20 for ; Thu, 12 May 2016 09:50:26 +0000 (UTC) (envelope-from rrs@netflix.com) Received: by mailman.ysv.freebsd.org (Postfix) id AF78AB37FF3; Thu, 12 May 2016 09:50:26 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF129B37FF2 for ; Thu, 12 May 2016 09:50:26 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mail-pf0-x22b.google.com (mail-pf0-x22b.google.com [IPv6:2607:f8b0:400e:c00::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 834F11A1F for ; Thu, 12 May 2016 09:50:26 +0000 (UTC) (envelope-from rrs@netflix.com) Received: by mail-pf0-x22b.google.com with SMTP id y69so29004333pfb.1 for ; Thu, 12 May 2016 02:50:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=from:subject:message-id:date:to:mime-version; bh=f824EXdyJ4J+GzOpz6zvfOS0zLGjCT/rEN9D7ZtLW+I=; b=nOpY/aU4cM8up6uOyuqlbW6SC+igU1vb4mpPOf+NE2e0FlaXozPPzfNqrc3p1RBVpr DZYEbViJfJ5i98pw915k+zseYAfBr6obhvOxCp2x1D5vQzlBjhctCIMM6t46aaUfZrwz MXGpzs0B36bfvZj9BFkKYk2k6lxzWByR4sTLo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:message-id:date:to:mime-version; bh=f824EXdyJ4J+GzOpz6zvfOS0zLGjCT/rEN9D7ZtLW+I=; b=G0KAmvxV6Nrf7YBBORvZSrS5Z2y18nksMdF2t4tIEcxNuGTisywwOU1vssO+zGa8Bm 9YTxs4CmmoHvkC6mtIPXA/CJSoIMBp4YMJz0aqYB7QxZ+1nN9mA4gHavIm/vawyB0h+f OE0CFxcvE50x3QOXlPfqjucK2EyzC1fxbkr54WXrYdRh+Hd46GNHRDQpbQcqu/p4GB+r 1dGjP9NjIaU1wI48a9i/SVdALjomHRKCv/ZkbeyIE+LvN9bSi7Fev/28O8zKmPsPlG/t PyX1Co8bbX+HzJLqqyy7NzFS/52+dMh5vdvXdAgpIr+AJ2H4rxr21TCT3epbmw5fPWU+ NcGw== X-Gm-Message-State: AOPr4FVClwJTmbi2eVMqa01YOGhGM9DeXZoiAm3OigbsD38U0+tccduPEfklaFM0B2x3U4Qt X-Received: by 10.98.18.80 with SMTP id a77mr12190297pfj.27.1463046626009; Thu, 12 May 2016 02:50:26 -0700 (PDT) Received: from lgml-rrs-2.corp.netflix.com ([69.53.231.133]) by smtp.gmail.com with ESMTPSA id y2sm18377340pfi.39.2016.05.12.02.50.23 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 12 May 2016 02:50:24 -0700 (PDT) From: Randall Stewart Subject: Consider this code in tcp_output.c and tcp_input.c Message-Id: Date: Thu, 12 May 2016 05:50:18 -0400 To: FreeBSD Transports Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2016 09:50:26 -0000 All: Here are a couple blocks of code from input/output ************************************ tcp_input.c: /* * Segment received on connection. * Reset idle time and keep-alive timer. * XXX: This should be done after segment * validation to ignore broken/spoofed segs. */ tp->t_rcvtime =3D ticks; tcp_output.c: /* * Determine length of data that should be transmitted, * and flags that will be used. * If there is some data or critical controls (SYN, RST) * to send, then transmit; otherwise, investigate further. */ idle =3D (tp->t_flags & TF_LASTIDLE) || (tp->snd_max =3D=3D = tp->snd_una); if (idle && ticks - tp->t_rcvtime >=3D tp->t_rxtcur) cc_after_idle(tp); ************************************** Now this works just fine if you are the sender of the data.. i.e. we go = idle, and then=20 you call send(=85) to the peer. However what happens if say you are a CDN? Your client goes idle, its gotten everything.. for some period of time, = maybe its playing out its play-buffer or what ever.. Then it sends in a request, "please send me a large block of data=94? You then start to stream out your 2Meg block of data=85. Since you received a request at tp->t_rcvtime to send you a big block of = data the timestamp is current. So you blast out 2Meg using your old cwnd/ssthresh and send a nice = line-rate burst=85 which of course hits tail drops and other fun :-) I believe we need to have something like: *************************** if ((tp->snd_max =3D=3D tp->and_una) && ((ticks - tp->t_rcvtime) >=3D = tp->t_rxtcur))) { cc_after_idle(tp) } tp->t_rcvtime =3D ticks; *********************************** added so we catch both sides of the equation.. R -------- Randall Stewart rrs@netflix.com 803-317-4952 From owner-freebsd-transport@freebsd.org Thu May 12 17:32:28 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3F46B388C5 for ; Thu, 12 May 2016 17:32:28 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id A28C71B89 for ; Thu, 12 May 2016 17:32:28 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id A1DADB388C4; Thu, 12 May 2016 17:32:28 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F2C5B388C3 for ; Thu, 12 May 2016 17:32:28 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 923701B88 for ; Thu, 12 May 2016 17:32:25 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 9F5471780D; Thu, 12 May 2016 10:32:19 -0700 (PDT) Date: Thu, 12 May 2016 10:32:19 -0700 From: hiren panchasara To: Randall Stewart Cc: FreeBSD Transports Subject: Re: Consider this code in tcp_output.c and tcp_input.c Message-ID: <20160512173219.GG51099@strugglingcoder.info> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="reI/iBAAp9kzkmX4" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2016 17:32:28 -0000 --reI/iBAAp9kzkmX4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/12/16 at 05:50P, Randall Stewart via freebsd-transport wrote: > All: >=20 > Here are a couple blocks of code from input/output > ************************************ > tcp_input.c: >=20 > /* > * Segment received on connection. > * Reset idle time and keep-alive timer. > * XXX: This should be done after segment > * validation to ignore broken/spoofed segs. > */ > tp->t_rcvtime =3D ticks; >=20 > tcp_output.c: >=20 > /* > * Determine length of data that should be transmitted, > * and flags that will be used. > * If there is some data or critical controls (SYN, RST) > * to send, then transmit; otherwise, investigate further. > */ > idle =3D (tp->t_flags & TF_LASTIDLE) || (tp->snd_max =3D=3D tp->s= nd_una); > if (idle && ticks - tp->t_rcvtime >=3D tp->t_rxtcur) > cc_after_idle(tp); >=20 > ************************************** >=20 > Now this works just fine if you are the sender of the data.. i.e. we go i= dle, and then=20 > you call send(?) to the peer. >=20 > However what happens if say you are a CDN? >=20 > Your client goes idle, its gotten everything.. for some period of time, m= aybe its playing out > its play-buffer or what ever.. >=20 > Then it sends in a request, "please send me a large block of data?? >=20 > You then start to stream out your 2Meg block of data?. >=20 > Since you received a request at tp->t_rcvtime to send you a big block of = data > the timestamp is current. >=20 > So you blast out 2Meg using your old cwnd/ssthresh and send a nice line-r= ate > burst? which of course hits tail drops and other fun :-) >=20 > I believe we need to have something like: > *************************** > if ((tp->snd_max =3D=3D tp->and_una) && ((ticks - tp->t_rcvtime) >=3D tp-= >t_rxtcur))) { > cc_after_idle(tp) > } > tp->t_rcvtime =3D ticks; > *********************************** > added so we catch both sides of the equation.. Great catch! I always thought we were covered on both sides but clearly thats not the case. BTW, we are very conservative in after idle case on our CC side which I tried to fix: https://reviews.freebsd.org/D5124 (which got abandoned for other things but you get the idea...) Cheers, Hiren --reI/iBAAp9kzkmX4 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJXNL4gXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lGT0H/jXEnK8jhrxovloWoAQysmTL rCyFGF5o8fyBqZ2BHKAtm8xpSDqoMn15C+gLf3WSfioe+gi/iRTG2NrxOOSRaBvx EMBNjU8COaO8JHO2Rxn+pqagUWPHIracS/zrj+ldh++3YXVf39Xpm7y0VqR8cjD/ AWaI2TP/Fe9HCzNoMQcG51yiWX4hWCyuh7UL1k2j/GjxjVD/UY/R4SCDu4nzBQWI +mTdb2LTEPig3rY1qWePwYobR2lriZGG7YQxRM/oQg3xwcICSHGm0tf9aneTGMwM dWassbJ55WOqzAUG+wRgNaOEDV2mhBfEaM2EaJzJxknXnZhs6r4NFncbT3HFN+A= =kub8 -----END PGP SIGNATURE----- --reI/iBAAp9kzkmX4-- From owner-freebsd-transport@freebsd.org Fri May 13 17:36:40 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 727D6B392C3 for ; Fri, 13 May 2016 17:36:40 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 6269D13A8 for ; Fri, 13 May 2016 17:36:40 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id 5DC73B392C2; Fri, 13 May 2016 17:36:40 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B0C7B392C1 for ; Fri, 13 May 2016 17:36:40 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 418BB13A6 for ; Fri, 13 May 2016 17:36:39 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 723B9170C0 for ; Fri, 13 May 2016 10:36:33 -0700 (PDT) Date: Fri, 13 May 2016 10:36:33 -0700 From: hiren panchasara To: transport@FreeBSD.org Subject: Abrupt reset sent instead of retransmitting a lost packet Message-ID: <20160513173633.GG44085@strugglingcoder.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="64j1qyTOoGvYcHb1" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 May 2016 17:36:40 -0000 --64j1qyTOoGvYcHb1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable https://people.freebsd.org/~hiren/pcaps/tcp_weird_reset.txt Something we saw in the wild on 10.2ish systems (server and client both). The most interesting thing can be seen at the end of the file. 3298737767:3298739215 gets lost, client tells us about it via a bunch of dupacks with SACK info. It SACKs all the outstanding data but this one missing packet. We (server) never retransmits that missing packet but rather decide to send a Reset after 0.312582ms. Which somehow causes client to pause for 75secs. (which might be another issue and not particularly important for this discussion.) What could cause this behavior of sending a reset instead of retransmitting a lost packet?=20 Cheers, Hiren --64j1qyTOoGvYcHb1 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJXNhCZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/ll9wH/jMku5AsfNer9G+owLpgIbIy +8z+A2csNJELmOd497IQfkkTvCtRO3xnzxFeV8uMer7ju+KiGX1xhVzEsEoQcyI1 y0qNel0zzhnnuUWisivBCnxA+rNhKRVd/tW7Vd3sX83Xjz9PaJZEvZqfwU8dTjsW fHN8ORU2b9eo5JL7SN/D5vknspFbkPdIOnDtKbQ0hODHuOW/Z1/D69R63h4aGikC Fm0eQRux41aQQ//eBQncm1BIQ4hJGb/1bzC8TgQxl7Xhd15WPv8WpMNGizUPVIj0 DI7JEuJnYYcs2bBvz+UFZ5tw+L7Zx+tOpvFg7dtJJEfkbx/P6aEdiTGj18cpZlQ= =lH+D -----END PGP SIGNATURE----- --64j1qyTOoGvYcHb1--