From owner-freebsd-questions@FreeBSD.ORG Tue Feb 14 04:07:40 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2BE5D16A422 for ; Tue, 14 Feb 2006 04:07:40 +0000 (GMT) (envelope-from jbell@stelesys.com) Received: from stelesys.com (web3.stelesys.com [63.175.100.43]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9006543D46 for ; Tue, 14 Feb 2006 04:07:39 +0000 (GMT) (envelope-from jbell@stelesys.com) Received: from c-71-199-184-251.hsd1.ga.comcast.net ([71.199.184.251] helo=[192.168.0.148]) by stelesys.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.60 (FreeBSD)) (envelope-from ) id 1F8rTW-0008Lr-OD; Mon, 13 Feb 2006 23:07:39 -0500 Message-ID: <43F1578A.8060803@stelesys.com> Date: Mon, 13 Feb 2006 23:07:38 -0500 From: Jerry Bell User-Agent: Thunderbird 1.5 (Windows/20051201) MIME-Version: 1.0 To: Jerry Bell References: <1716.209.134.164.18.1139835495.squirrel@www.stelesys.com> <9CB4E9E3-EE93-4E65-AD74-0ACC9B3C64FF@mac.com> <4026.209.134.164.18.1139861525.squirrel@www.stelesys.com> <1DFDCA44-74AC-475A-96A9-0E3AD5B492C4@mac.com> <43F13D9F.5040606@stelesys.com> In-Reply-To: <43F13D9F.5040606@stelesys.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-questions@freebsd.org Subject: Re: Help with strange web server problem X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2006 04:07:40 -0000 Looks like it's still an issue, so I'd say the firewall issue is still in play. If there is not a firewall/proxy in place, are there any known issues with IPFW (or anything else with FBSD) that could cause this behavior? Jerry Bell wrote: > Charles - thank you for your excellent investigation! I'm pretty sure > that my colo provider isn't running a firewall (I've asked them not > to, anyhow). I am running IPFW on that box, with the standard "allow > tcp from any to any established" followed by the "allow tcp any to > my_ip 80 setup". I've done that on other servers without it being a > problem like this. I'm going to have the colo double check for router > acl's or something like that in the morning. > > Since this is such an intermittent problem, I can't yet say that it's > fixed, but I ran with the "disks being idled" theory and wrote a small > script that creates a file and deletes a file every minute, and since > that's been running, I've not seeing the issue repeat - but then this > is not a very repeatable problem. > > Thanks again for your great assistance. > > Jerry > > > Charles Swiger wrote: >> On Feb 13, 2006, at 3:12 PM, Jerry Bell wrote: >>> I didn't want to spam the link out, but it's www.musiclodge.com. I >>> will >>> gather the capture data from working and non working sessions and >>> send it >>> out. >> >> Well, I can confirm the behavior you've described. >> >> It looks somewhat like a stateful firewall or is in the way and is >> generating an RST, even while your webserver tries to generate a >> response. However, once the firewall sees the outbound traffic, it >> seems to create a dynamic rule which lets the traffic from subsequent >> connections through: >> >> 5-pan# tcpdump -tnXs 0 host www.musiclodge.com >> tcpdump: verbose output suppressed, use -v or -vv for full protocol >> decode >> listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes >> IP 199.103.21.238.50740 > 63.175.100.44.80: S >> 2282569549:2282569549(0) win 65535 > 0,nop,nop,timestamp 1159441862 0> >> 0x0000: 4510 003c 4653 4000 4006 7328 c767 15ee >> E..> 0x0010: 3faf 642c c634 0050 880d 3f4d 0000 0000 >> ?.d,.4.P..?M.... >> 0x0020: a002 ffff 815f 0000 0204 05b4 0103 0300 >> ....._.......... >> 0x0030: 0101 080a 451b adc6 0000 0000 ....E....... >> IP 63.175.100.44.80 > 199.103.21.238.50740: S >> 2634350592:2634350592(0) ack 2282569550 win 65535 >> 0x0000: 4500 0028 0000 4000 2506 d49f 3faf 642c >> E..(..@.%...?.d, >> 0x0010: c767 15ee 0050 c634 9d05 0000 880d 3f4e >> .g...P.4......?N >> 0x0020: 5012 ffff 03bc 0000 0000 0000 0000 1b60 >> P..............` >> 0x0030: 2678 &x >> IP 199.103.21.238.50740 > 63.175.100.44.80: . ack 1 win 65535 >> 0x0000: 4510 0028 4655 4000 4006 733a c767 15ee >> E..(FU@.@.s:.g.. >> 0x0010: 3faf 642c c634 0050 880d 3f4e 9d05 0001 >> ?.d,.4.P..?N.... >> 0x0020: 5010 ffff 03bd 0000 P....... >> >> 3-way handshake is completed here, next traffic should be from my >> machine making the "GET /", request, but instead your machine sends >> another ACK: >> >> IP 63.175.100.44.80 > 199.103.21.238.50740: S >> 2238145710:2238145710(0) ack 2282569550 win 65535 > 1460,nop,wscale 1,nop,nop,timestamp 1453026167 1159441862> >> 0x0000: 4500 003c 57fa 4000 3206 6f91 3faf 642c >> E..> 0x0010: c767 15ee 0050 c634 8567 64ae 880d 3f4e >> .g...P.4.gd...?N >> 0x0020: a012 ffff 9cdb 0000 0204 05b4 0103 0301 >> ................ >> 0x0030: 0101 080a 569b 6b77 451b adc6 9345 1153 >> ....V.kwE....E.S >> >> Interesting that the previous ack had no TCP options set, whereas >> this one does include a timestamp in response. >> >> IP 199.103.21.238.50740 > 63.175.100.44.80: . ack 396204883 win 65535 >> >> 0x0000: 4510 0034 4656 4000 4006 732d c767 15ee >> E..4FV@.@.s-.g.. >> 0x0010: 3faf 642c c634 0050 880d 3f4e 9d05 0001 >> ?.d,.4.P..?N.... >> 0x0020: 8010 ffff 8157 0000 0101 080a 451b adc7 >> .....W......E... >> 0x0030: 569b 6b77 V.kw >> >> Where did sequence # 396204883 come from? And your side follows up >> with a pair of connection resets, and a normal ACK packet, too. >> >> IP 63.175.100.44.80 > 199.103.21.238.50740: R >> 2634350593:2634350593(0) win 0 >> 0x0000: 4500 0028 b6f6 4000 3206 10a9 3faf 642c >> E..(..@.2...?.d, >> 0x0010: c767 15ee 0050 c634 9d05 0001 0000 0000 >> .g...P.4........ >> 0x0020: 5004 0000 cb24 0000 0000 0000 0000 f3fa >> P....$.......... >> 0x0030: 5489 T. >> IP 63.175.100.44.80 > 199.103.21.238.50740: R >> 2634350593:2634350593(0) win 0 >> 0x0000: 4500 0028 4bfc 4000 3206 7ba3 3faf 642c >> E..(K.@.2.{.?.d, >> 0x0010: c767 15ee 0050 c634 9d05 0001 0000 0000 >> .g...P.4........ >> 0x0020: 5004 0000 cb24 0000 0000 0000 0000 abb8 >> P....$.......... >> 0x0030: c9be .. >> IP 63.175.100.44.80 > 199.103.21.238.50740: S >> 2238145710:2238145710(0) ack 2282569550 win 65535 > 1460,nop,wscale 1,nop,nop,timestamp 1453026467 1159441862> >> 0x0000: 4500 003c 3a9d 4000 3206 8cee 3faf 642c >> E..<:.@.2...?.d, >> 0x0010: c767 15ee 0050 c634 8567 64ae 880d 3f4e >> .g...P.4.gd...?N >> 0x0020: a012 ffff 9baf 0000 0204 05b4 0103 0301 >> ................ >> 0x0030: 0101 080a 569b 6ca3 451b adc6 bdd6 d7c9 >> ....V.l.E....... >> >> ...and my side closes, too. Something is badly confused. >> >> IP 199.103.21.238.50740 > 63.175.100.44.80: R >> 2282569550:2282569550(0) win 0 >> 0x0000: 4500 0028 465a 4000 4006 7345 c767 15ee >> E..(FZ@.@.sE.g.. >> 0x0010: 3faf 642c c634 0050 880d 3f4e 0000 0000 >> ?.d,.4.P..?N.... >> 0x0020: 5004 0000 a0cf 0000 P....... >> >> ------------------- >> >> When I repeat the connection attempt a few seconds later: >> >> IP 199.103.21.238.50743 > 63.175.100.44.80: S 262625798:262625798(0) >> win 65535 >> 0x0000: 4510 003c 46c8 4000 4006 72b3 c767 15ee >> E..> 0x0010: 3faf 642c c637 0050 0fa7 5a06 0000 0000 >> ?.d,.7.P..Z..... >> 0x0020: a002 ffff 815f 0000 0204 05b4 0103 0300 >> ....._.......... >> 0x0030: 0101 080a 451b b055 0000 0000 ....E..U.... >> IP 63.175.100.44.80 > 199.103.21.238.50743: S 362624500:362624500(0) >> ack 262625799 win 65535 > 1453058903 1159442517> >> 0x0000: 4500 003c e034 4000 3206 e756 3faf 642c >> E..<.4@.2..V?.d, >> 0x0010: c767 15ee 0050 c637 159d 35f4 0fa7 5a07 >> .g...P.7..5...Z. >> 0x0020: a012 ffff 169b 0000 0204 05b4 0103 0301 >> ................ >> 0x0030: 0101 080a 569b eb57 451b b055 55d6 9ceb >> ....V..WE..UU... >> IP 199.103.21.238.50743 > 63.175.100.44.80: . ack 1 win 65535 >> >> 0x0000: 4510 0034 46c9 4000 4006 72ba c767 15ee >> E..4F.@.@.r..g.. >> 0x0010: 3faf 642c c637 0050 0fa7 5a07 159d 35f5 >> ?.d,.7.P..Z...5. >> 0x0020: 8010 ffff 8157 0000 0101 080a 451b b055 >> .....W......E..U >> 0x0030: 569b eb57 V..W >> >> 3-way handshake finishes OK, this time your initial ACK is OK and >> includes a timestamp back. >> >> IP 199.103.21.238.50743 > 63.175.100.44.80: P 1:17(16) ack 1 win >> 65535 >> 0x0000: 4510 0044 46cd 4000 4006 72a6 c767 15ee >> E..DF.@.@.r..g.. >> 0x0010: 3faf 642c c637 0050 0fa7 5a07 159d 35f5 >> ?.d,.7.P..Z...5. >> 0x0020: 8018 ffff 8167 0000 0101 080a 451b b05d >> .....g......E..] >> 0x0030: 569b eb57 4745 5420 2f20 4854 5450 2f31 >> V..WGET./.HTTP/1 >> 0x0040: 2e30 0d0a .0.. >> >> Here was my GET, which you then ACK.... >> >> IP 63.175.100.44.80 > 199.103.21.238.50743: . ack 17 win 33304 >> >> 0x0000: 4500 0034 052a 4000 3206 c269 3faf 642c >> E..4.*@.2..i?.d, >> 0x0010: c767 15ee 0050 c637 159d 35f5 0fa7 5a17 >> .g...P.7..5...Z. >> 0x0020: 8010 8218 be99 0000 0101 080a 569b eced >> ............V... >> 0x0030: 451b b05d db4e 4827 E..].NH' >> IP 199.103.21.238.50743 > 63.175.100.44.80: P 17:19(2) ack 1 win >> 65535 >> 0x0000: 4510 0036 46ce 4000 4006 72b3 c767 15ee >> E..6F.@.@.r..g.. >> 0x0010: 3faf 642c c637 0050 0fa7 5a17 159d 35f5 >> ?.d,.7.P..Z...5. >> 0x0020: 8018 ffff 8159 0000 0101 080a 451b b05e >> .....Y......E..^ >> 0x0030: 569b eced 0d0a V..... >> >> ...followed by a correct data response. >> >> IP 63.175.100.44.80 > 199.103.21.238.50743: . 1:1449(1448) ack 19 win >> 33304 >> 0x0000: 4500 05dc c865 4000 3206 f985 3faf 642c >> E....e@.2...?.d, >> 0x0010: c767 15ee 0050 c637 159d 35f5 0fa7 5a19 >> .g...P.7..5...Z. >> 0x0020: 8010 8218 2e5f 0000 0101 080a 569b ed01 >> ....._......V... >> 0x0030: 451b b05e 4854 5450 2f31 2e31 2032 3030 >> E..^HTTP/1.1.200 >> 0x0040: 204f 4b0d 0a44 6174 653a 204d 6f6e 2c20 >> .OK..Date:.Mon,. >> [ ... ] >> >> Check your firewall, or see whether your ISP or whoever has put a >> HTTP reverse proxy in place which is breaking these connections. >> >> ---Chuck >> > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org"