From owner-freebsd-transport@freebsd.org Sun Oct 18 00:37:42 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37767A17B2E for ; Sun, 18 Oct 2015 00:37:42 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1F5D77AD for ; Sun, 18 Oct 2015 00:37:42 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id 1C7A2A17B2D; Sun, 18 Oct 2015 00:37:42 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C17AA17B2C for ; Sun, 18 Oct 2015 00:37:42 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.strugglingcoder.info", Issuer "mail.strugglingcoder.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B2047AC for ; Sun, 18 Oct 2015 00:37:41 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id C294C10A9A2; Sat, 17 Oct 2015 17:37:40 -0700 (PDT) Date: Sat, 17 Oct 2015 17:37:40 -0700 From: hiren panchasara To: Randall Stewart Cc: FreeBSD Transports Subject: dupack counter processing (was: Re: The trouble with sack..) Message-ID: <20151018003740.GE87252@strugglingcoder.info> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ExIO7AL6tcaLjJBF" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2015 00:37:42 -0000 --ExIO7AL6tcaLjJBF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 10/07/15 at 12:17P, Randall Stewart via freebsd-transport wrote: >=20 > 2) When we recognize a dup-ack we *will not* recognize it if for example = if the rwnd changes even > if new SACK information is reported in the sack blocks. This is due t= o the fact that in non-SACK you don?t > (on purpose) recognize ACK?s where the window changed (since you can?= t really tell if its a > plain window update or a dup-ack).. This means we occasionally miss = out > on stroking the dup-ack counter and getting out of recovery.... Just learned that linux triggers fast recovery right away when it receives a dupack with SACK info that covers at least 3 packets (essentially indicating that 3 packets made it through successfully after a loss/drop event). Here, 3 is tcprexmtthresh for us. This also follows rfc6675. (IsLost()). Cheers, Hiren --ExIO7AL6tcaLjJBF Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJWIunRXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lM9MH/1ZkZJPiVDimJHEKhgzIcwmg /PsqKP1ATuN4fK06aJJGle8YvZFx1EHB+wtAUvMHSAkUH/ZAcZ2w2GV1FA/f8Rt3 npitd1opabQBD/wfwiZxWMCdsYQXf/iWaUVs3lzKpsa5kfAl4HOgPfyL3EhODZpD n9THvUDpSAdTVtwQ6hXouFIqahQMzrnYfBnBiqhfU7HL8VvuPQWJMTWOqLLr4riU lIyobY+e8ndT0Xbp6ogdkgpODjE3cW45AwzaM/UzQfqj/GC42pqim7L01uSP6en9 M57O21rl5iMPFS+tImHcNv2QnY6Ax/2UnpRasWGukvaRx0WzNQ8HRngAStkLTrM= =yLhA -----END PGP SIGNATURE----- --ExIO7AL6tcaLjJBF-- From owner-freebsd-transport@freebsd.org Wed Oct 21 01:32:02 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B90FA1977F for ; Wed, 21 Oct 2015 01:32:02 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 85D0B3D2 for ; Wed, 21 Oct 2015 01:32:02 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id 8568CA1977E; Wed, 21 Oct 2015 01:32:02 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85042A1977D for ; Wed, 21 Oct 2015 01:32:02 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.strugglingcoder.info", Issuer "mail.strugglingcoder.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 75E5F3D1; Wed, 21 Oct 2015 01:32:02 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 9204A10A89C; Tue, 20 Oct 2015 18:32:01 -0700 (PDT) Date: Tue, 20 Oct 2015 18:32:01 -0700 From: hiren panchasara To: transport@FreeBSD.org Cc: rrs@FreeBSD.org Subject: Help me understand this SACK code in tcp_output() Message-ID: <20151021013201.GD28288@strugglingcoder.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="x4pBfXISqBoDm8sr" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 01:32:02 -0000 --x4pBfXISqBoDm8sr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline tcp_output() starts out with following two vars set to zero: sack_rxmit = 0; sack_bytes_rxmt = 0; And if we are in recovery and trying to send from holes, i.e. if ((tp->t_flags & TF_SACK_PERMIT) && IN_FASTRECOVERY(tp->t_flags) && (p = tcp_sack_output(tp, &sack_bytes_rxmt))) { And if we end up with if (len > 0), we set sack_rxmit = 1; A little below is following code which I do not understand: /* If sack_rxmit is true we are retransmitting from the scoreboard * in which case len is already set. */ if (sack_rxmit == 0) { if (sack_bytes_rxmt == 0) ...... else { } } What I do not understand is how can we get into the 'else' part. AFAICT, sack_rxmit: when TRUE, we are retransmitting from the scoreboard sack_bytes_rxmt: number of retransmitted bytes from the scoreboard. Now, if sack_rxmit is false, how can sack_bytes_rxmt be > 0? OR is there something I am missing in the code OR how SACK works? Cheers, Hiren --x4pBfXISqBoDm8sr Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJWJusOXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/l+3QH/0edSDsmGh1SsXQg/TA6dQdg P0Sj6qMFTWtAwibmiDR0wbjZgxVGkXVXBRisN2eNHRIPeyil65K72cEF2W2Q5LUA PFaV5eQrc7ctGBbouQJcldm2gSS16OzAwwdJGRSy7TfeUsMvCRTHQI0AIN53pBRw Q7I2Tuzlh6txcmvHzkNCeSHbJprugPjHR8pEszOFG3njyHa3mNpkGW7OEvtNOmcO L6VIdu8PSPxMvR+33xhmC90qmHOs1PQoN6FiVdxWWEQKUOgtBvcBQpXUQUEe5l/k TXFFNdVzTHhdzMgAw1s8PQB6NuNdejR5etdPCXO8vo7AA553JIGydbIJG062pUo= =Q6gB -----END PGP SIGNATURE----- --x4pBfXISqBoDm8sr-- From owner-freebsd-transport@freebsd.org Wed Oct 21 23:22:12 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D264A1BB42 for ; Wed, 21 Oct 2015 23:22:12 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 03D411DF5 for ; Wed, 21 Oct 2015 23:22:12 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id 022D3A1BB41; Wed, 21 Oct 2015 23:22:12 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBFB6A1BB40 for ; Wed, 21 Oct 2015 23:22:11 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.strugglingcoder.info", Issuer "mail.strugglingcoder.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C10911DF4 for ; Wed, 21 Oct 2015 23:22:11 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 7DFFA10A767 for ; Wed, 21 Oct 2015 16:22:10 -0700 (PDT) Date: Wed, 21 Oct 2015 16:22:10 -0700 From: hiren panchasara To: transport@FreeBSD.org Subject: Re: Correct inflight/pipe calculation Message-ID: <20151021232210.GL28288@strugglingcoder.info> References: <20151007172610.GA42742@strugglingcoder.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="nrM5Z5VIJgwP9LWp" Content-Disposition: inline In-Reply-To: <20151007172610.GA42742@strugglingcoder.info> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 23:22:12 -0000 --nrM5Z5VIJgwP9LWp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 10/07/15 at 10:26P, hiren panchasara wrote: > Randall and I have been poking at different ways to improve FreeBSD > tcp's reaction to loss. One of the major issue we found is that we do > not use information provided by SACK efficiently even though we do keep > the SACK scoreboard well in shape. Knowing amount of data in flight can > be really crucial and can help us use available capacity of the path > more efficiently. We currently do not have an accurate way of knowing > this information. >=20 > For example, inside tcp_do_segment(), while processing duplicate acks, > we try to compute amount of data inflight with: > awnd =3D (tp->snd_nxt - tp->snd_fack) + > tp->sackhint.sack_bytes_rexmit; >=20 > Which is incorrect as it doesn't take into account whats been already > sacked by the receiver. > There are definitely other places in the stack where we do this > incorrectly. >=20 > RFC 6675 provides guidance on how to implement calculations for > bytes in flight at any point in time. Randall and I came to a conclusion > that following can provide us inflight information almost(!) accurately > with least amount of code changes: >=20 > pipe =3D snd_max - snd_una - sackhint.sacked_bytes + sackhint.sack_bytes_= rexmit; >=20 > here, > snd_max: highest sequence number sent > snd_una: lowest sequence number sent but not yet cumulatively acked > sacked_bytes: total bytes sacked by receiver reported via SACK holes > sack_bytes_rexmit: total bytes retransmitted from holes in this recovery > period >=20 > Only missing piece in FreeBSD is sackd_bytes. This is basically total > bytes sacked by the receiver and it can be extracted from SACK holes > reported by the receiver. The approach we've decided to take is pretty > simple: we already process each ACK with sack holes in=20 > tcp_sack_doack() and extract sack blocks out of it. We'd now also track > this new variable there which keeps track of total sacked bytes > reported. >=20 > The downside with this approach is: > There is no persistent information about sacked bytes. We recalculate > it every time we get an ACK with sack holes in it. So if, for any > reason, receiver decides to drop sack info than we get incorrect > value for inflight. This may be also true when there are more holes but > receiver can only report 3 at a time. >=20 > I have actual code that I've been testing and if people see no major prob= lem > with this approach, I can put it up for review in phabricator. Well, I didn't receive any replies so here it is: https://reviews.freebsd.org/D3971 Please take a look at that. Cheers, Hiren --nrM5Z5VIJgwP9LWp Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJWKB4fXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/l6qoH/097Rf3e5qQmlsjESRVr8+lh 8GQ+jaZ1kUh+YRcjq7yJpPhsyY8gdGYkzfDR9/rYrP2obuCT1f7IeoUftVZvUAtM sx0/NP6laikisEr8po+ndBRttJ/nSzKPYb0gx67Txhv0ydiNDzY/FQEgMjp9z9m5 0vJX/9hwrl7edUaotpbm8pugIkrLcAW2QDJuCPZws7/W6ZeQjBtoczwcw745pI8c K+mgbAYChEJOsW0tlF2Wp23BxjLoTPWr5ixIgGLVCQbCA0EPIG9CYYyN2aB4NRbC kIxwIZ9rt+3HrauV22yIyoJUkgEdrLNA256dU53qQkKhOr5cc3B6djPIFrH58ak= =Xze1 -----END PGP SIGNATURE----- --nrM5Z5VIJgwP9LWp--