From owner-freebsd-net@FreeBSD.ORG Thu Mar 29 14:04:10 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B6D51065673 for ; Thu, 29 Mar 2012 14:04:10 +0000 (UTC) (envelope-from hunter@comsys.com.ua) Received: from mail.ice-tech.com.ua (mail.ice-tech.com.ua [77.120.117.100]) by mx1.freebsd.org (Postfix) with ESMTP id 157828FC08 for ; Thu, 29 Mar 2012 14:04:09 +0000 (UTC) Received: from [94.247.224.226] (helo=hunters-MacBook-Pro.local) by mail.ice-tech.com.ua with esmtpa (Exim 4.77 (FreeBSD)) (envelope-from ) id 1SDFQF-0004Y5-9b for freebsd-net@freebsd.org; Thu, 29 Mar 2012 16:29:51 +0300 Message-ID: <4F7463CF.8010206@comsys.com.ua> Date: Thu, 29 Mar 2012 16:29:51 +0300 From: Sergey Smitienko User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0) Gecko/20120327 Thunderbird/11.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 94.247.224.226 X-SA-Exim-Mail-From: hunter@comsys.com.ua X-SA-Exim-Scanned: No (on mail.ice-tech.com.ua); SAEximRunCond expanded to false Subject: FreeBSD 9.0 generates incorrect SEC/ACK numbers under load X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 14:04:10 -0000 Hello. I've run into a problem with a web server runing FreeBSD 9.0/amd64. What I believe is happening, is what server loses track of correct SEQ/ACK numbers on some connections. Here is an example: 15:20:00.347514 IP (tos 0x68, ttl 123, id 1181, offset 0, flags [DF], proto TCP (6), length 52) 93.72.14.220.49239 > 193.178.147.113.80: Flags [S], cksum 0x6995 (correct), seq 3881466934, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0 15:20:00.347526 IP (tos 0x10, ttl 254, id 28065, offset 0, flags [DF], proto TCP (6), length 44) 193.178.147.113.80 > 93.72.14.220.49239: Flags [S.], cksum 0x79fa (correct), seq 2151790680, ack 3881466935, win 0, options [mss 1460], length 0 15:20:00.361812 IP (tos 0x68, ttl 123, id 1183, offset 0, flags [DF], proto TCP (6), length 40) 93.72.14.220.49239 > 193.178.147.113.80: Flags [.], cksum 0x96c6 (correct), seq 3881466935, ack 2151790681, win 64240, length 0 15:20:00.361869 IP (tos 0x10, ttl 254, id 31305, offset 0, flags [DF], proto TCP (6), length 40) 193.178.147.113.80 > 93.72.14.220.49239: Flags [.], cksum 0x71b7 (correct), seq 2151790681, ack 3881466935, win 8192, length 0 Client sends "GET" request 15:20:48.236181 IP (tos 0x68, ttl 123, id 1353, offset 0, flags [DF], proto TCP (6), length 626) 93.72.14.220.49239 > 193.178.147.113.80: Flags [P.], cksum 0x7fc9 (correct), seq 3881466935:3881467521, ack 2151790681, win 64240, length 586 and then the "ping-pong" starts: 15:20:48.236198 IP (tos 0x0, ttl 254, id 63530, offset 0, flags [DF], proto TCP (6), length 40) 193.178.147.113.80 > 93.72.14.220.49239: Flags [.], cksum 0x8a97 (correct), seq 2991748588, ack 1985077892, win 8760, length 0 15:20:48.255998 IP (tos 0x68, ttl 123, id 1357, offset 0, flags [DF], proto TCP (6), length 40) 93.72.14.220.49239 > 193.178.147.113.80: Flags [.], cksum 0x947c (correct), seq 3881467521, ack 2151790681, win 64240, length 0 15:20:48.256015 IP (tos 0x0, ttl 254, id 53518, offset 0, flags [DF], proto TCP (6), length 40) 193.178.147.113.80 > 93.72.14.220.49239: Flags [.], cksum 0x8a97 (correct), seq 2991748588, ack 1985077892, win 8760, length 0 15:20:48.276084 IP (tos 0x68, ttl 123, id 1360, offset 0, flags [DF], proto TCP (6), length 40) 93.72.14.220.49239 > 193.178.147.113.80: Flags [.], cksum 0x947c (correct), seq 3881467521, ack 2151790681, win 64240, length 0 15:20:48.276099 IP (tos 0x0, ttl 254, id 42983, offset 0, flags [DF], proto TCP (6), length 40) 193.178.147.113.80 > 93.72.14.220.49239: Flags [.], cksum 0x8a97 (correct), seq 2991748588, ack 1985077892, win 8760, length 0 15:20:48.290914 IP (tos 0x68, ttl 123, id 1361, offset 0, flags [DF], proto TCP (6), length 40) 93.72.14.220.49239 > 193.178.147.113.80: Flags [.], cksum 0x947c (correct), seq 3881467521, ack 2151790681, win 64240, length 0 This happens on about 0.01% of connections. This tcpdump is recorded on the 193.178.147.113, before traffic hits the wire. So it's not a NIC fault. Server is running nginx and serving static content 200-500 request per second. Any ideas ? -- Sergey Smitienko