From owner-freebsd-net@FreeBSD.ORG Mon Mar 28 01:39:28 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11D781065672 for ; Mon, 28 Mar 2011 01:39:28 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id BC8AA8FC0C for ; Mon, 28 Mar 2011 01:39:27 +0000 (UTC) Received: by iyj12 with SMTP id 12so3900108iyj.13 for ; Sun, 27 Mar 2011 18:39:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:in-reply-to :message-id:references:user-agent:x-openpgp-key-id :x-openpgp-key-fingerprint:mime-version:content-type; bh=BvKaeZcyaShabrGSSQqKZziW0VGvdUrFRmnN3NDQmBE=; b=DSUnoMuSqt2qe43Jfp1sB3WVIb5o0FGktrlR0T7e6B+jwvKPraa/W2dveQVWwCUPe6 rZvZ8ML1qFCSrLjSqikVdSudo34mFQfsYNV01k/yPKmVohxybkidGJPUqJfGtr3xmNRs X3KB5ul2kCunNOxRuCROVBWOZfImaSY9RY7cA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=L9jB+hRwY4hK6duDLdJuXRwxLXldqYCXUzIo2KddNNvuzOlZIg3x34zuY286uW4shS HRQTtutMl+j87C1/yvyhtAWlsDw0rr1dFCXTsZAHj5FmWr5VMGLuWzWLDoLbYWAaF6PL u0EE1h+svV0INPNxnNjTPbW1FSzfGM957nCzg= Received: by 10.231.52.209 with SMTP id j17mr3401887ibg.163.1301276367039; Sun, 27 Mar 2011 18:39:27 -0700 (PDT) Received: from disbatch.dataix.local (adsl-99-181-153-110.dsl.klmzmi.sbcglobal.net [99.181.153.110]) by mx.google.com with ESMTPS id u9sm2584749ibe.2.2011.03.27.18.39.22 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 27 Mar 2011 18:39:23 -0700 (PDT) Sender: "J. Hellenthal" Date: Sun, 27 Mar 2011 21:38:57 -0400 From: "J. Hellenthal" To: Stefan `Sec` Zehl In-Reply-To: <20110326224340.GB23803@ice.42.org> Message-ID: References: <4D8B99B4.4070404@FreeBSD.org> <201103250825.10674.jhb@freebsd.org> <20110325194109.GB25392@ice.42.org> <201103251640.16147.jhb@freebsd.org> <20110326140212.GB45402@ice.42.org> <20110326224340.GB23803@ice.42.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="80310268-1666522583-1301276363=:9813" Cc: freebsd-net@freebsd.org Subject: Re: The tale of a TCP bug X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2011 01:39:28 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --80310268-1666522583-1301276363=:9813 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 26 Mar 2011 18:43, sec@ wrote: > Hi, > >> On Fri, Mar 25, 2011 at 16:40 -0400, John Baldwin wrote: >>> And the problem is that the code that uses 'adv' to determine if it >>> sound send a window update to the remote end is falsely succeeding due >>> to the overflow causing tcp_output() to 'goto send' but that it then >>> fails to send any data because it thinks the remote window is full? > > On a whim I wanted to find out, how often that overflow is triggered in > normal operation, and whipped up a quick counter-sysctl. > > --- sys/netinet/tcp_output.c.org 2011-01-04 19:27:00.000000000 +0100 > +++ sys/netinet/tcp_output.c 2011-03-26 18:49:30.000000000 +0100 > @@ -87,6 +87,11 @@ > extern struct mbuf *m_copypack(); > #endif > > +VNET_DEFINE(int, adv_neg) = 0; > +SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, adv_neg, CTLFLAG_RD, > + &VNET_NAME(adv_neg), 1, > + "How many times adv got negative"); > + > VNET_DEFINE(int, path_mtu_discovery) = 1; > SYSCTL_VNET_INT(_net_inet_tcp, OID_AUTO, path_mtu_discovery, CTLFLAG_RW, > &VNET_NAME(path_mtu_discovery), 1, > @@ -573,6 +578,10 @@ > long adv = min(recwin, (long)TCP_MAXWIN << tp->rcv_scale) - > (tp->rcv_adv - tp->rcv_nxt); > > + if(min(recwin, (long)TCP_MAXWIN << tp->rcv_scale) < > + (tp->rcv_adv - tp->rcv_nxt)) > + adv_neg++; > + > if (adv >= (long) (2 * tp->t_maxseg)) > goto send; > if (2 * adv >= (long) so->so_rcv.sb_hiwat) > > I booted my main (web/shell) box with (only) this patch: > > 11:36PM up 3:50, 1 user, load averages: 2.29, 1.51, 0.73 > net.inet.tcp.adv_neg: 2466 > > That's approximately once every 5 seconds. That's way more often than I > suspected. > > CU, > Sec > With this patch applied with John's on a 32-bit box I can repeatedly bump this sysctl with an SSL connection to another destination. Doesn't seem to matter what the destination is. curl -q https://www.changeip.com/ip.asp It also bumps in SSL connections to other protocols too. This behavior does not seem to be happening with non-SSL connections. Attached is a script that I am using to monitor the sysctl here just for reference. L = Last value C = Current value D = Difference I = Log interval S = Seconds since last change * = marked changed line /bin/sh ./adv_neg_mon.sh 7 |tee -a adv_neg.log [...] L:41 C:41 D:0 I:7 S:7.000000e+01 L:41 C:41 D:0 I:7 S:7.700000e+01 L:41 C:43 D:2 I:7 S:8.400000e+01 * L:43 C:88 D:45 I:7 S:7.000000e+00 * - -- Regards, J. Hellenthal (0x89D8547E) JJH48-ARIN -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x89D8547E iQEcBAEBAgAGBQJNj+a7AAoJEJBXh4mJ2FR+VssIAI7QSUUb6jvZdMWxxVGPpr6o vPGDqPfWxNcih4D5SZxJJtsslnunpAcOjSWK8YGvOCINt8XhexVOSklyHuyvjIWd 4ijywngx5H2RT22c6wTdNPOfsZzoBkvLZZ2mj2cUF1ISxrvgy5syMp/TnANE3kul Mqf29HA8t3qYQCfb6zuFoWGdYI5Ahfsks4rljZJy/5bRQfNceJwBjUGnSlL0651m Bl4GpcNWA0fbuJeUgEzIK6mOpNdoI+PrZv6GEG7LErLaVtr+43gET/YITuGv1jY3 dlQ1WkHZSnaG/S7vpWbb2W/cuJ8ak6esbM74x8KakiOnLeJgy0MYK8oqYJyN3aI= =l+iW -----END PGP SIGNATURE----- --80310268-1666522583-1301276363=:9813 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=adv_neg_mon.sh Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: adv_neg monitor Content-Disposition: attachment; filename=adv_neg_mon.sh IyEvYmluL3NoDQoNCnRyYXAgJ2V4aXQgMScgMg0KDQpVUERBVEU9JDEgOzog JHtVUERBVEU6PTV9DQoNCndoaWxlIHRydWU7IGRvDQoJTlZBTD0kKHN5c2N0 bCAtbiBuZXQuaW5ldC50Y3AuYWR2X25lZykNCglpZiBbIC16ICIkTFZBTCIg XTsgdGhlbg0KCQlMVkFMPSR7TlZBTH0NCglmaQ0KCWlmIFsgIiROVkFMIiAt Z3QgIiRMVkFMIiBdOyB0aGVuDQoJCWVjaG8gIkw6JExWQUwgQzokTlZBTCBE OiQoKCR7TlZBTH0tJHtMVkFMfSkpIEk6JHtVUERBVEV9IFM6JChwcmludGYg JWUgJHtVU0VDU30pICoiDQoJCVVTRUNTPSR7VVBEQVRFfQ0KCWVsc2UNCgkJ ZWNobyAiTDokTFZBTCBDOiROVkFMIEQ6JCgoJHtOVkFMfS0ke0xWQUx9KSkg SToke1VQREFURX0gUzokKHByaW50ZiAlZSAke1VTRUNTfSkiDQoJCVVTRUNT PSQoKCR7VVNFQ1N9KyR7VVBEQVRFfSkpDQoJZmkNCglMVkFMPSR7TlZBTH0N CglzbGVlcCAkVVBEQVRFDQpkb25lDQo= --80310268-1666522583-1301276363=:9813--