From owner-freebsd-net@FreeBSD.ORG  Tue Aug 10 13:51:39 2010
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id AA0B01065677;
	Tue, 10 Aug 2010 13:51:39 +0000 (UTC)
	(envelope-from sethj@greatbaysoftware.com)
Received: from portcityhosting.com (edge.tidalhosting.net [64.140.243.92])
	by mx1.freebsd.org (Postfix) with ESMTP id 2B35F8FC17;
	Tue, 10 Aug 2010 13:51:38 +0000 (UTC)
Received: from sjeacopello ([173.14.128.81]) by portcityhosting.com with
	MailEnable ESMTP; Tue, 10 Aug 2010 09:51:36 -0400
X-WatchGuard-Mail-Exception: Allow
From: "Seth Jeacopello" <sethj@greatbaysoftware.com>
To: "'Andre Oppermann'" <andre@freebsd.org>
References: <B5F0E510AC5548FD83D57BE077F66866@sjeacopello><4C5DE0D5.7090802@freebsd.org><D12EF2FC899A47289CE235D18F219F9E@sjeacopello>
	<4C6117D5.2070207@freebsd.org>
Date: Tue, 10 Aug 2010 09:51:36 -0400
Message-ID: <A9CA9CB9C850450BB077BF2B1BAE4967@sjeacopello>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
In-Reply-To: <4C6117D5.2070207@freebsd.org>
X-WatchGuard-AntiVirus: part scanned. clean action=allow
X-ME-Bayesian: 0.000000
Cc: freebsd-net@freebsd.org
Subject: RE: Server sporadically sending unexpected RST to Client
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Aug 2010 13:51:39 -0000

Interestingly the patch you provided is almost identical to what we put in
place on the system to trace the issue earlier.  Without having merged the
patch yet, let me give you what we do have from that run:

Aug  6 11:15:27 kernel: Entering syncache_add 

Aug  6 11:15:27 kernel: Entering syncache_expand 

Aug  6 11:15:27 kernel: TCP: [10.174.50.35]:52872 to [10.174.80.242]:389
tcpflags 0x18<PUSH,ACK>; syncache_expand: Just before syncache_socket() 

Aug  6 11:15:27 kernel: Entering syncache_socket 

Aug  6 11:15:27 kernel: Entering syncache_expand 

Aug  6 11:15:27 kernel: TCP: [10.174.50.35]:52872 to [10.174.80.242]:389
tcpflags 0x10<ACK>; syncache_expand: Just before syncache_socket() 

Aug  6 11:15:27 kernel: Entering syncache_socket 

Aug  6 11:15:27 kernel: TCP: [10.174.50.35]:52872 to [10.174.80.242]:389;
syncache_socket: Socket create failed in_pcbconnect: 48 

Aug  6 11:15:27 kernel: TCP: [10.174.50.35]:52872 to [10.174.80.242]:389
tcpflags 0x10<ACK>; tcp_input: Listen socket: Socket allocation failed due
to limits or memory shortage, sending RST


Also, I wanted to note again that the tcpdumps we've grabbed are showing
that the packets are going across the wire in order, however, debug logging
of the issue on the Server side has shown they appear to be processed out of
order (with the PSH, ACK being processed before the final ACK of the
handshake).


--Seth