From owner-freebsd-stable@FreeBSD.ORG Sun Aug 29 19:10:44 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25A191065693 for ; Sun, 29 Aug 2010 19:10:44 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 8C1D58FC1B for ; Sun, 29 Aug 2010 19:10:43 +0000 (UTC) Received: (qmail 45682 invoked from network); 29 Aug 2010 19:08:39 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 29 Aug 2010 19:08:39 -0000 Message-ID: <4C7AB0B2.9020003@freebsd.org> Date: Sun, 29 Aug 2010 21:10:42 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Thunderbird/3.1.2 MIME-Version: 1.0 To: Mike Tancsa References: <201008291409.o7TE9QCJ082862@lava.sentex.ca> <4C7A7DD4.9020803@freebsd.org> <201008291716.o7THGxcF083744@lava.sentex.ca> In-Reply-To: <201008291716.o7THGxcF083744@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: syncache errors X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Aug 2010 19:10:44 -0000 On 29.08.2010 19:16, Mike Tancsa wrote: > At 11:33 AM 8/29/2010, Andre Oppermann wrote: >> On 29.08.2010 16:09, Mike Tancsa wrote: >>> After upgrading to a recent STABLE, I have been seeing the following sporadic errors >>> >>> Aug 28 04:15:15 smarthost2 kernel: TCP: [xx.yy.165.120]:53617 to [xx.yy.164.50]:25; syncache_socket: >>> Socket create failed due to limits or memory shortage >>> >>> this is with >>> FreeBSD 8.1-STABLE #7: Wed Aug 25 15:32:05 EDT 2010 >>> and the previous kernel was from July 20th. >>> >>> The odd thing is that the error is triggered from a RELENG_6 host only it would seem. I noticed >>> there are a number of syncache and tcp updates to RELENG_8, is it possible this is related ? I dont >>> have any specific tweaks in /etc/sysctl.conf >>> >>> other than >>> net.inet.tcp.log_in_vain: 1 >>> net.inet.udp.log_in_vain: 1 >> >> The log_in_vain sysctl would cause the logging of syncache errors >> as well (net.inet.tcp.log_debug). This was a POLA violation and >> I've separated them on August 27 on 8-stable. So if you update >> you won't see them anymore. That doesn't change the fact that >> the socket create failed though. >> >> If it only happens from a RELENG_6 box it probably is a problem >> with port re-usage by RELENG_6. The difficulty is that sonewconn() >> fails and doesn't return an error code. Hence it may be one of >> listen queue limits reached, memory shortage or a problem with >> inserting the inpcb into the hash lists. > > Actually, I think I might have found the issue. I was focusing on the "memory" part not the limits. > The 'RELENG_6 only' is just a fluke as thats a box that sends a lot of email to this particular > server. It turns out, sendmail was rate limiting the server > sm-mta[25923]: deferring connections on daemon IPv4: 8 per second > and somehow syncache is aware/logs this now where as it did not before ? When sendmail is deferring the connections it should not show up in the syncache logs. The accept() by sendmail is much later than when the socket for a new connection is created in syncache. Here it points to the limits in the listen queue. Maybe sendmail is getting behind in accepting new connections and the listen queue is overflowing. -- Andre